Systems and methods for updating continuous image alignment of separate cameras

ABSTRACT

A system for updating continuous image alignment of separate cameras identifies a previous alignment matrix associated with a previous frame pair captured at one or more previous timepoints by a reference camera and a match camera. The previous alignment matrix is based on visual correspondences in the previous frame pair. The system also identifies a current matrix associated with a current frame pair captured at one or more current timepoints by the reference camera and the match camera. The current matrix is based on visual correspondences in the current frame pair. The system also identifies a difference value associated with the reference camera or the match camera relative to the one or more previous timepoints and the one or more current timepoints. The system also generates an updated alignment matrix by using the previous alignment matrix, the current matrix, and the difference value as inputs.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/936,312, filed on Jul. 22, 2020, and entitled “SYSTEMS AND METHODSFOR UPDATING CONTINUOUS IMAGE ALIGNMENT OF SEPARATE CAMERAS”, theentirety of which is incorporated herein by reference.

BACKGROUND

Mixed-reality (MR) systems, including virtual-reality andaugmented-reality systems, have received significant attention becauseof their ability to create truly unique experiences for their users. Forreference, conventional virtual-reality (VR) systems create a completelyimmersive experience by restricting their users' views to only a virtualenvironment. This is often achieved, in VR systems, through the use of ahead-mounted device (HMD) that completely blocks any view of the realworld. As a result, a user is entirely immersed within the virtualenvironment. In contrast, conventional augmented-reality (AR) systemscreate an augmented-reality experience by visually presenting virtualobjects that are placed in or that interact with the real world.

As used herein, VR and AR systems are described and referencedinterchangeably. Unless stated otherwise, the descriptions herein applyequally to all types of mixed-reality systems, which (as detailed above)includes AR systems, VR reality systems, and/or any other similar systemcapable of displaying virtual objects.

Some MR systems include one or more cameras and utilize images and/ordepth information obtained using the camera(s) to provide pass-throughviews of a user's environment to the user. A pass-through view can aidusers in avoiding disorientation and/or safety hazards whentransitioning into and/or navigating within a mixed-reality environment.Pass-through views may also enhance user views in low visibilityenvironments. For example, mixed-reality systems configured with longwavelength thermal imaging cameras may facilitate visibility in smoke,haze, fog, and/or dust. Likewise, mixed-reality systems configured withlow light imaging cameras facilitate visibility in dark environmentswhere the ambient light level is below the level required for humanvision.

An MR system may provide pass-through views in various ways. Forexample, an MR system may present raw images captured by the camera(s)of the MR system to a user. In other instances, an MR system may modifyand/or reproject captured image data to correspond to the perspective ofa user's eye to generate pass-through views. An MR system may modifyand/or reproject captured image data to generate a pass-through viewusing depth information for the captured environment obtained by the MRsystem (e.g., using a depth system of the MR system, such as a time offlight camera, a rangefinder, stereoscopic depth cameras, etc.). In someinstances, an MR system utilizes one or more predefined depth values togenerate pass-through views (e.g., by performing planar reprojection).

In some instances, pass-through views generated by modifying and/orreprojecting captured image data may at least partially correct fordifferences in perspective brought about by the physical separationbetween a user's eyes and the camera(s) of the MR system (known as the“parallax problem,” “parallax error,” or, simply “parallax”). Suchpass-through views/images may be referred to as “parallax-correctedpass-through” views/images. By way of illustration, parallax-correctedpass-through images may appear to a user as though they were captured bycameras that are co-located with the user's eyes.

MR systems are often used in combination with other devices that arephysically independent from and/or untethered to the MR system (e.g.,controllers, instruments, etc.). For example, a user may operate an MRsystem while also operating a handheld/wearable device that includes adevice camera. The device camera may be configured to be directed atand/or capture portions of the environment that are within the field ofview of the MR system, such that both the device camera and cameras ofthe MR system concurrently capture portions of the environment.

User experiences may be enhanced by providing composite pass-throughimages on an MR system that implement camera imagery captured by aseparate device camera aligned with and overlaid on camera imagerycaptured by the camera(s) of the MR system. However, because MR systemcameras and separate device cameras move independently of one another,accurately aligning the camera imagery of the separate cameras togenerate such composite pass-through images is associated with manychallenges, particularly in real-time and/or near-real-timeimplementations.

For at least the foregoing reasons, there is an ongoing need and desirefor improved techniques and systems for facilitating continuous imagealignment of separate cameras.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

Disclosed embodiments include systems and methods for updatingcontinuous image alignment of separate cameras.

Some embodiments include methods implemented by a computing system, suchas a head-mounted display (HMD), in which the computing system performsvarious acts, including an act of identifying a previous alignmentmatrix associated with a previous frame pair captured at one or moreprevious timepoints by a reference camera and a match camera. Theprevious alignment matrix is based on visual correspondences betweenimages of the previous frame pair.

The computing system also performs an act of identifying a currentmatrix associated with a current frame pair captured at one or morecurrent timepoints by the reference camera and the match camera. Thecurrent matrix is based on visual correspondences between images of thecurrent frame pair.

The computing system also performs an act of identifying a differencevalue associated with the reference camera or the match camera relativeto the one or more previous timepoints and the one or more currenttimepoints.

The computing system also performs an act of generating an updatedalignment matrix by using the previous alignment matrix, the currentmatrix, and the difference value as inputs for generating the updatedalignment matrix.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims. Features of the present invention will become more fullyapparent from the following description and appended claims or may belearned by the practice of the invention as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof the subject matter briefly described above will be rendered byreference to specific embodiments which are illustrated in the appendeddrawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting inscope, embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates an example mixed-reality system that may include orbe used to implement disclosed embodiments;

FIG. 2 illustrates an example head-mounted display (HMD) and a userinstrument that include various cameras that may facilitate thedisclosed embodiments, including a reference camera and a match camera;

FIG. 3 illustrates an example of capturing an environment with areference camera and a match camera;

FIG. 4 illustrates an example of feature matching between a referenceframe and a match frame;

FIG. 5 illustrates an example of unprojecting the feature matchesidentified according to FIG. 4 ;

FIG. 6 illustrates an example of identifying a base matrix using theunprojected feature matches of FIG. 5 ;

FIG. 7 illustrates an example of unprojecting a set of pixels of areference frame to generated 3D points;

FIG. 8 illustrates an example of generating modified 3D points byapplying the base matrix of FIG. 6 to the 3D points of FIG. 7 ;

FIG. 9 illustrates an example of projecting the modified 3D points ofFIG. 8 ;

FIG. 10 illustrates an example of a composite image that includesreference frame pixels mapped to corresponding match frame pixels in anoverlap region;

FIG. 11 illustrates an example of capturing the environment from FIG. 3with the reference camera and the match camera at subsequent timepoints;

FIG. 12 illustrates an example of unprojecting a set of pixels of anupdated reference frame to generate 3D points;

FIG. 13 illustrates an example of generating modified 3D points byapplying a reference camera transformation matrix to the 3D points ofFIG. 12 ;

FIG. 14 illustrates an example of generating modified 3D points byapplying the base matrix to the modified 3D points of FIG. 13 ;

FIG. 15 illustrates an example of generating modified 3D points byapplying a match camera transformation matrix to the 3D points of FIG.14 ;

FIG. 16 illustrates an example of projecting the modified 3D points ofFIG. 15 ;

FIG. 17 illustrates an example of a composite image that includesupdated reference frame pixels mapped to updated corresponding matchframe pixels in an overlap region;

FIG. 18 illustrates an example of feature matching between the updatedreference frame and the updated match frame;

FIG. 19 illustrates an example of unprojecting the feature matchesidentified according to FIG. 18 ;

FIG. 20 illustrates an example of identifying an updated matrix usingthe unprojected feature matches of FIG. 19 ;

FIGS. 21 and 22 illustrate an example of generating an aligned updatedmatrix by modifying the updated matrix from FIG. 20 using inertialtracking data associated with the reference camera and the match camera;

FIG. 23A illustrates an example of generating an alignment matrix usinga base matrix and the aligned updated matrix from FIGS. 21 and 22 ;

FIG. 23B illustrates an example of generating a subsequent alignmentmatrix using the alignment matrix from FIG. 23A and a subsequent alignedupdated matrix;

FIG. 24 illustrates an example of generating a composite image using areference frame, a match frame, and a motion model;

FIG. 25 illustrates an example flow diagram depicting acts associatedwith generating a motion model configured to facilitate mapping of a setof pixels of a reference frame captured by a reference camera to acorresponding set of pixels of a match frame captured by a match camera;

FIG. 26 illustrates an example flow diagram depicting acts associatedwith facilitating continuous image alignment of two cameras;

FIG. 27 illustrates an example of a reference camera and a match cameracapturing an environment at different timepoints;

FIG. 28 illustrates example alignment matrices that may be associatedwith frame pairs captured by the reference camera and the match camera;

FIG. 29 illustrates an example of difference values associated with thereference camera and the match camera at different timepoints;

FIG. 30 illustrates an example of generating an updated alignment matrixusing an aligned current matrix, a previous alignment matrix, and one ormore difference values as inputs;

FIG. 31 illustrates examples of difference values that may be associatedwith the reference camera and the match camera at different timepoints;

FIG. 32 illustrates an example plot of a smoothness function accordingto an implementation of the present disclosure;

FIG. 33 illustrates an example flow diagram depicting acts associatedwith updating continuous image alignment of a reference camera and amatch camera; and

FIG. 34 illustrates an example computer system that may include and/orbe used to perform disclosed embodiments.

DETAILED DESCRIPTION

Disclosed embodiments include systems and methods for updatingcontinuous image alignment of separate cameras.

Some embodiments include methods implemented by a computing system, suchas a head-mounted display (HMD), in which the computing system performsvarious acts, including an act of identifying a previous alignmentmatrix associated with a previous frame pair captured at one or moreprevious timepoints by a reference camera and a match camera. Theprevious alignment matrix is based on visual correspondences betweenimages of the previous frame pair. The computing system also performs anact of identifying a current matrix associated with a current frame paircaptured at one or more current timepoints by the reference camera andthe match camera. The current matrix is based on visual correspondencesbetween images of the current frame pair. The computing system alsoperforms an act of identifying a difference value associated with thereference camera or the match camera relative to the one or moreprevious timepoints and the one or more current timepoints. Thecomputing system also performs an act of generating an updated alignmentmatrix by using the previous alignment matrix, the current matrix, andthe difference value as inputs for generating the updated alignmentmatrix.

Examples of Technical Benefits, Improvements, and Practical Applications

Those skilled in the art will recognize, in view of the presentdisclosure, that at least some of the disclosed embodiments may addressvarious shortcomings associated with conventional approaches forfacilitating continuous image alignment of separate cameras. Thefollowing section outlines some example improvements and/or practicalapplications provided by the disclosed embodiments. It will beappreciated, however, that the following are examples only and that theembodiments described herein are in no way limited to the exampleimprovements discussed herein.

In some implementations, providing a motion model based on referencecamera and match camera transformation matrices (for poses) and analignment matrix enables a system to map pixels of a reference frame toa match frame (and/or vice versa) as the positions of the referencecamera and the match camera change relative to one another over time.For example, the reference camera and match camera transformationmatrices may be updated/determined as the poses of the reference cameraand the match camera change, which enables the motion model to accountfor updates in the positions of the cameras.

In some implementations, the alignment matrix is a 3D rotational matrix,which may correspond to reference camera and match camera transformationmatrices based on inertial tracking data obtained by inertialmeasurement units (IMU) associated with the reference camera and thematch camera, respectively. Accordingly, at least some implementationsof the present disclosure facilitate simple concatenation of thealignment matrix with the camera transformation matrices based on IMUdata to generate a motion model.

Furthermore, in some instances, the alignment matrix may beupdated/determined based on visual correspondences between imagescaptured by the reference camera and the match camera, which may enablethe motion model to ameliorate the effects of IMU drift and/or parallax.

In addition, an alignment matrix may be generated by fusing (e.g.,blending) a base matrix (based on visual correspondences for a baseframe pair) with an updated matrix (based on visual correspondences foran updated frame pair obtained subsequent to the base frame pair), whichmay address noise/imprecision that may be associated with using visualcorrespondences to determine an alignment matrix.

Still furthermore, a system may intelligently determine whether or howto blend a previous alignment matrix with an updated/current matrix togenerate an updated alignment matrix based on difference valuesassociated with the reference camera and/or the match camera relative tothe timepoints at which a previous frame pair and an updated frame pairwere obtained. Such implementations may enable systems to selectivelyignore past alignment matrices upon indications that they have becomeunreliable (e.g., where sufficient motion has occurred or time haselapsed since the previous alignment matrix was computed).

A motion model of the present disclosure may enable a system tocontinuously map pixels of a reference frame captured by a referencecamera to corresponding pixels of a match frame captured by a matchcamera. Thus, a motion model may enable a system to generate compositepass-through images that include imagery of the reference camera (e.g.,a separate device camera) aligned with and overlaid on imagery of thematch camera (e.g., one or more head-mounted display (HMD) camera).Providing such composite pass-through images may enable users to readilyidentify which portion of an environment a separate device (and/ordevice camera) is directed toward, which may enable users to directinput and/or actions toward portions of the environment in an accurateand/or rapid manner. Such composite pass-through images may bebeneficial in various applications, such as, for example, gamingenvironments, medical/dental operations/training, first respondertraining/activities, and/or others.

Although the present disclosure focuses, in some respects, onfacilitating continuous image alignment of a match camera of an HMD anda reference camera of a user instrument (e.g., separate user device) toprovide composite pass-through views for display on the HMD, it will beappreciated, in view of the present disclosure, that the principlesdisclosed herein are applicable to any implementation that involvesproviding continuous image alignment between any number of separatecameras.

Having just described some of the various high-level features andbenefits of the disclosed embodiments, attention will now be directed toFIGS. 1 through 33 . These Figures illustrate various conceptualrepresentations, architectures, methods, and supporting illustrationsrelated to the disclosed embodiments. The disclosure will then turn toFIG. 34 , which presents an example computer system that may includeand/or be used to facilitate the disclosed principles.

Example Mixed-Reality Systems and HMDs

Attention will now be directed to FIG. 1 , which illustrates an exampleof a head-mounted device (HMD) 100. HMD 100 can be any type ofmixed-reality system 100A (MR system), including a VR system 100B or anAR system 100C. It should be noted that while a substantial portion ofthis disclosure is focused, in some respects, on the use of an HMD, theembodiments are not limited to being practiced using only an HMD. Thatis, any type of system can be used, even systems entirely removed orseparate from an HMD. As such, the disclosed principles should beinterpreted broadly to encompass any type of scanning scenario ordevice. Some embodiments may even refrain from actively using a scanningdevice themselves and may simply use the data generated by the scanningdevice. For instance, some embodiments may at least be partiallypracticed in a cloud computing environment.

FIG. 1 illustrates HMD 100 as including sensor(s) 150, includingscanning sensor(s) 105 and other sensors, such as accelerometer(s) 155,gyroscope(s) 160, compass(es) 165. The ellipsis 170 conveys that thesensor(s) 150 depicted in FIG. 1 are illustrative only and non-limiting.For instance, in some implementations, an HMD 100 includes otherinteroceptive and/or exteroceptive sensors not explicitly illustrated inFIG. 1 , such as eye tracking systems, radio-based navigation systems,microphones, and/or other sensing apparatuses. In some implementations,an HMD 100 includes fewer sensors than those depicted in FIG. 1 .

The accelerometer(s) 155, gyroscope(s) 160, and compass(es) 165 areconfigured to measure inertial tracking data. Specifically, theaccelerometer(s) 155 is/are configured to measure acceleration, thegyroscope(s) 160 is/are configured to measure angular velocity data, andthe compass(es) 165 is/are configured to measure heading data. In someinstances, an HMD 100 utilizes the inertial tracking components thereofto obtain three degree of freedom (3DOF) pose data associated with theHMD (e.g., where visual tracking data, described below, is unavailable,unreliable, and/or undesired). As used herein, 3DOF refers to position(e.g., rotation) information associated with rotational axes about threeperpendicular directional axes (e.g., pitch, yaw, and roll).

The inertial tracking components/system of the HMD 100 (i.e., theaccelerometer(s) 155, gyroscope(s) 160, and compass(es) 165) may operatein concert with a visual tracking system to form a head tracking systemthat generates pose data for the HMD 100. In some instances, a visualtracking system includes one or more cameras (e.g., head trackingcameras) that capture image data of an environment (e.g., environment175). In some instances, the HMD 100 obtains visual tracking data basedon the images captured by the visual tracking system, such as objectswithin the environment that may provide an anchor for determiningmovement of the HMD 100 relative to the environment.

For example, visual-inertial Simultaneous Location and Mapping (SLAM) inan HMD 100 fuses (e.g., with a pose filter) visual tracking dataobtained by one or more cameras (e.g., head tracking cameras) withinertial tracking data obtained by the accelerometer(s) 155,gyroscope(s) 160, and compass(es) 165 to estimate six degree of freedom(6DOF) positioning (i.e., pose) of the HMD 100 in space and in realtime. 6DOF refers to positioning/velocity information associated withthree perpendicular directional axes and the three rotational axes(often referred to as pitch, yaw, and roll) about each of the threeperpendicular directional axes (often referred to as x, y, and z).

Unless otherwise specified, any reference herein to a “pose” or arelated term describing positioning and/or orientation may refer to 3DOFor 6DOF pose.

The visual tracking system of an HMD 100, in some instances, includes astereo pair of head tracking images that is configured to obtain depthmaps of the user's environment (e.g., environment 175) to provide visualmapping of the user's environment (e.g., by maintaining a surface meshof the environment, or any other 3D representation of the environment).The HMD 100 may utilize the visual mapping data of the environment toaccurately display virtual content with respect to the user'senvironment. Visual mapping data may also enable location sharingbetween users in a shared mixed-reality environment.

In some instances, the visual tracking system(s) of an HMD 100 (e.g.,head tracking cameras) is/are implemented as one or more dedicatedcameras. In other instances, the visual tracking system(s) is/areimplemented as part of a camera system that performs other functions(e.g., as part of one or more cameras of the scanning sensor(s) 105,described hereinbelow).

The scanning sensor(s) 105 comprise any type of scanning or camerasystem, and the HMD 100 can employ the scanning sensor(s) 105 to scanenvironments, map environments, capture environmental data, and/orgenerate any kind of images of the environment. For example, in someinstances, the HMD 100 is configured to generate a 3D representation ofthe real-world environment or generate a “pass-through” visualization.Scanning sensor(s) 105 may comprise any number or any type of scanningdevices, without limit.

In accordance with the disclosed embodiments, the HMD 100 may be used togenerate a parallax-corrected pass-through visualization of the user'senvironment. A “pass-through” visualization refers to a visualizationthat presents one or more images captured by cameras to a user,regardless of whether the HMD 100 is included as a part of an AR systemor a VR system. To generate this passthrough visualization, the HMD 100may use its scanning sensor(s) 105 to scan, map, or otherwise record itssurrounding environment, including any objects in the environment, andto pass that data on to the user to view. In many cases, thepassed-through data is modified to reflect or to correspond to aperspective of the user's pupils. The perspective may be determined byany type of eye tracking technique. In some instances, as the cameramodules are not telecentric with the user's eyes, the perspectivedifference between the user's eyes and the camera modules may becorrected to provide parallax-corrected pass-through visualizations.

To generate a parallax-corrected passthrough image, the scanningsensor(s) 105 may rely on its cameras (e.g., visible light camera(s)110, low light camera(s) 115, thermal imaging camera(s) 120, UVcamera(s) 125, or any other type of camera) to obtain one or more rawimages of the environment (e.g., environment 175). In some instances,these raw images may also be used to determine depth data detailing thedistance from the sensor to any objects captured by the raw images(e.g., a z-axis range or measurement). Once these raw images areobtained, then a depth map can be computed from the depth data embeddedor included within the raw images, and passthrough images can begenerated (e.g., one for each pupil) using the depth map for anyreprojections.

As used herein, a “depth map” details the positional relationship anddepths relative to objects in the environment. Consequently, thepositional arrangement, location, geometries, contours, and depths ofobjects relative to one another can be determined. From the depth maps(and possibly the raw images), a 3D representation of the environmentcan be generated. Unless otherwise specified, the terms “depth map” and“disparity map” are used interchangeably herein.

Relatedly, from the passthrough visualizations, a user will be able toperceive what is currently in the user's environment without having toremove or reposition the HMD 100. Furthermore, the disclosed passthroughvisualizations may also enhance the user's ability to view objectswithin his/her environment (e.g., by displaying additional environmentalconditions that may not have been detectable by a human eye).

It should be noted that while a portion of this disclosure focuses ongenerating “a” passthrough image, the implementations described hereinmay generate a separate passthrough image for each one of the user'seyes. That is, two passthrough images may be generated concurrently withone another. Therefore, while frequent reference is made to generatingwhat seems to be a single passthrough image, the implementationsdescribed herein are actually able to simultaneously generate multiplepassthrough images.

In some embodiments, scanning sensor(s) 105 include cameras of variousmodalities, such as visible light camera(s) 110, low light camera(s)115, thermal imaging camera(s) 120, Near Infrared (NIR) Cameras (in the800 nm to 2 um range), and/or potentially (though not necessarily)ultraviolet (UV) cameras 125. The ellipsis 130 demonstrates how anyother type of camera or camera system (e.g., depth cameras, time offlight cameras, etc.) may be included among the scanning sensor(s) 105.As an example, a camera structured to detect mid-infrared wavelengthsmay be included within the scanning sensor(s) 105.

Generally, a human eye is able to perceive light within the so-called“visible spectrum,” which includes light (or rather, electromagneticradiation) having wavelengths ranging from about 380 nanometers (nm) upto about 740 nm. In some instances, the visible light camera(s) 110include monochrome cameras structured to capture light photons withinthe visible spectrum (and/or, in some instances, the infrared spectrum).In some instances, the visible light camera(s) 110 include red, green,blue (RGB) cameras structured to capture light photons within thevisible spectrum (and/or, in some instances, the infrared spectrum). Insome implementations, visible light camera(s) 110 are complementarymetal-oxide-semiconductor (CMOS) type cameras, though other camera typesmay be used as well (e.g., charge coupled devices, CCD).

Visible light camera(s) 110 may be implemented as stereoscopic cameras,meaning that the fields of view of two or more visible light cameras 110at least partially overlap with one another. With this overlappingregion, images generated by the visible light camera(s) 110 can be usedto identify disparities between certain pixels that commonly representan object captured by both images. Disparities are typically measuredafter applying rectification to the stereo pair of images such thatcorresponding pixels in the images that commonly represent an object inthe environment are aligned along scanlines. After rectification,corresponding pixels in the different images that commonly represent anobject in the environment only differ in one dimension (e.g., thedirection of the scanlines, such as the horizontal direction). Theone-dimensional difference between the coordinates of correspondingpixels in their respective images of the stereo pair of imagesrepresents the disparity value for the object represented by thecorresponding pixels.

Based on these pixel disparities, the embodiments are able to determinedepths for objects located within the overlapping region (i.e.“stereoscopic depth matching,” “stereo depth matching,” or simply“stereo matching”). The depths for the objects/3D points of theenvironment located within the overlapping region may be represented aspixels of a depth map. As such, the visible light camera(s) 110 can beused to not only generate passthrough visualizations, but they can alsobe used to determine object depth. depth information about thereal-world environment surrounding the mixed-reality system may enablethe system to accurately present mixed-reality content (e.g., holograms)with respect to real-world objects. As an illustrative example, a depthsystem may obtain depth information for a real-world table positionedwithin a real-world environment. The mixed-reality system is then ableto render and display a virtual figurine accurately positioned on thereal-world table such that the user perceives the virtual figurine asthough it were part of the user's real-world environment.

Those skilled in the art will recognize, in view of the presentdisclosure, that stereo matching may be performed on a stereo pair ofimages obtained by any type and/or combination of cameras. For example,an HMD 100 or other system may comprise any combination of visible lightcamera(s) 110, low light camera(s) 115, thermal imaging camera(s) 120,UV camera(s) 125, Near Infrared Red, and/or other cameras to capture astereo pair of images upon which to perform stereo matching (e.g., forthe overlapping region of the stereo pair of images).

In some instances, the low light camera(s) 115 are structured to capturevisible light and IR light. IR light is often segmented into threedifferent classifications, including near-IR, mid-IR, and far-IR (e.g.,thermal-IR). The classifications are determined based on the energy ofthe IR light. By way of example, near-IR has relatively higher energy asa result of having relatively shorter wavelengths (e.g., between about750 nm and about 1,000 nm). In contrast, far-IR has relatively lessenergy as a result of having relatively longer wavelengths (e.g., up toabout 30,000 nm). Mid-IR has energy values in between or in the middleof the near-IR and far-IR ranges. In some instances, the low lightcamera(s) 115 are structured to detect or be sensitive to IR light in atleast the near-IR range.

In some embodiments, the visible light camera(s) 110 and the low lightcamera(s) 115 operate in approximately the same overlapping wavelengthrange. In some cases, this overlapping wavelength range is between about400 nanometers and about 1,000 nanometers. Additionally, in someembodiments these two types of cameras are both silicon detectors.

In some instances, one distinguishing feature between these two types ofcameras is related to the illuminance conditions or illuminance range(s)in which they actively operate. In some cases, the visible lightcamera(s) 110 are low power cameras and operate in environments wherethe illuminance is between about 10 lux and about 100,000 lux (e.g., foran example commercial visible light camera), or rather, the illuminancerange begins at about 10 lux and increases beyond 10 lux. In contrast,the low light camera(s) 115 consume more power and operate inenvironments where the illuminance range is between overcast starlightand dusk lighting levels. In some instances, the device operates inenvironments between starlight conditions (e.g., about 1 milli-lux, fora typical commercial low light camera) and dusk conditions (e.g., about10 lux, for a typical commercial low light camera).

The thermal imaging camera(s) 120, in some instances, are structured todetect electromagnetic radiation or IR light in the far-IR (i.e.thermal-IR) range, though some implementations also enable the thermalimaging camera(s) 120 to detect radiation in the mid-IR range. Toclarify, the thermal imaging camera(s) 120 may be a long wave infraredimaging camera structured to detect electromagnetic radiation bymeasuring long wave infrared wavelengths. Often, the thermal imagingcamera(s) 120 detect IR radiation having wavelengths between about 8microns and 14 microns. Because the thermal imaging camera(s) 120 detectfar-IR radiation, the thermal imaging camera(s) 120 can operate, in someinstances, in any illuminance condition.

In some cases (though not necessarily all), the thermal imagingcamera(s) 120 include an uncooled thermal imaging sensor. An uncooledthermal imaging sensor uses a specific type of detector design that isbased on a bolometer, which is a device that measures the magnitude orpower of an incident electromagnetic wave/radiation. To measure theradiation, the bolometer uses a thin layer of absorptive material (e.g.,metal) connected to a thermal reservoir through a thermal link. Theincident wave strikes and heats the material. In response to thematerial being heated, the bolometer detects a temperature-dependentelectrical resistance. Changes to environmental temperature causechanges to the bolometer's temperature, and these changes can beconverted into an electrical signal to thereby produce a thermal imageof the environment. In accordance with at least some of the disclosedembodiments, the uncooled thermal imaging sensor is used to generate anynumber of thermal images. The bolometer of the uncooled thermal imagingsensor can detect electromagnetic radiation across a wide spectrum,spanning the mid-IR spectrum, the far-IR spectrum, and even up tomillimeter-sized waves.

The UV camera(s) 125 are structured to capture light in the UV range.The UV range includes electromagnetic radiation having wavelengthsbetween about 10 nm and about 400 nm. The disclosed UV camera(s) 125should be interpreted broadly and may be operated in a manner thatincludes both reflected UV photography and UV induced fluorescencephotography.

In some instances, visible light cameras are cameras that are used forcomputer vision to perform head tracking (e.g., as describedhereinabove). These cameras can detect visible light, or even acombination of visible and IR light (e.g., a range of IR light). In somecases, these cameras are global shutter devices with pixels being about3 μm in size.

Low light cameras, in some instances, are cameras that are sensitive tovisible light and near-IR. These cameras are larger and may have pixelsthat are about 5 μm in size or larger. These cameras are also sensitiveto wavelengths that silicon sensors are sensitive to, which wavelengthsare between about 350 nm to 1100 nm.

In some implementations, thermal/long wavelength IR devices (i.e.thermal imaging cameras) have pixel sizes that are about 10 μm or largerand detect heat radiated from the environment. These cameras may besensitive to wavelengths in the 8 μm to 14 μm range. Some embodimentsalso include mid-IR cameras configured to detect at least mid-IR light.These cameras often comprise non-silicon materials (e.g., InP-basedInGaAs cameras) that detect light in the 800 nm to 2 μm wavelengthrange.

Generally, the low light camera(s) 115, the thermal imaging camera(s)120, and the UV camera(s) 125 (if present) consume relatively more powerthan the visible light camera(s) 110. Therefore, when not in use, thelow light camera(s) 115, the thermal imaging camera(s) 120, and/or theUV camera(s) 125 are typically in the powered-down state in which thosecameras are either turned off (and thus consuming no power) or in areduced operability mode (and thus consuming substantially less powerthan if those cameras were fully operational). In contrast, the visiblelight camera(s) 110 are typically in the powered-up state in which thosecameras are by default fully operational.

It should be noted that any number of cameras may be provided on the HMD100 for each of the different camera types. That is, the visible lightcamera(s) 110 may include 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more than 10cameras. Often, however, the number of cameras is at least 2 so the HMD100 can perform stereoscopic depth matching, as described earlier.Similarly, the low light camera(s) 115, the thermal imaging camera(s)120, and the UV camera(s) 125 may each respectively include 1, 2, 3, 4,5, 6, 7, 8, 9, 10, or more than 10 corresponding cameras.

Continuous Image Alignment of Separate Cameras

Attention is now directed to FIG. 2 , which illustrates an examplehead-mounted display (HMD) 200 and a user instrument 250 that includevarious cameras that may facilitate the disclosed embodiments. The HMD200 may correspond, in at least some respects, to the HMD 100 describedhereinabove with reference to FIG. 1 . The user instrument 250 maycomprise any type of handheld and/or wearable device that is usable inconjunction with the HMD 200 (or another system associated with matchcamera(s) 215). For example, in some instances, a user instrument 250 isa controller, a medical/dental instrument, a first responder tool, etc.

The HMD 200 includes match camera(s) 215, and the user instrument 250includes a reference camera 260. The match camera(s) 215 and/or thereference camera 260 may be implemented as cameras of any modality(e.g., any combination of visible light camera(s) 110, low lightcamera(s) 115, thermal imaging camera(s) 120, UV camera(s) 125, NearInfrared Red, and/or other cameras). In some implementations, the matchcamera(s) 215 and the reference camera 260 share the same cameramodality, whereas in other implementations, the match camera(s) 215 andthe reference camera 260 have different camera modalities.

FIG. 2 depicts the HMD 200 with two match cameras 215. In someimplementations, the two match cameras 215 are substantially verticallyaligned the eyes of a user operating the HMD 200. In some instances, anHMD 200 with two match cameras 215 may enable the HMD 200 to provideper-eye parallax-corrected images (e.g., based on composite images, asdescribed hereinafter with reference to at least FIGS. 10 and 17 ).However, those skilled in the art will recognize, in view of the presentdisclosure, that the particular configuration of the HMD 200 and/or theuser instrument 250 depicted in FIG. 2 is illustrative only andnon-limiting. For example, in some instances, an HMD 200 includes one ormore than two match cameras 215, and/or a user instrument 250 includesmore than one reference camera 260.

FIG. 2 also illustrates that, in some instances, the HMD 200 includesother cameras 220 that may serve various functions, such as headtracking, hand/object tracking, video capture, etc. The HMD 200 alsoincludes displays 225A and 225B for displaying virtual content (e.g.,holograms, composite images, etc.) to a user wearing the HMD 200.

The HMD 200 of FIG. 2 includes a match camera inertial measurement unit(IMU) 235, and the user instrument 250 of FIG. 2 includes a referencecamera IMU 265. The match camera IMU 235 includes any combination ofaccelerometer(s) 155, gyroscope(s) 160, and/or compass(es) forgenerating inertial tracking data 240 (as described hereinabove). Insome instances, the match camera IMU 235 is mounted to the HMD 200 at afixed position relative to the match camera(s) 215, such that theinertial tracking data 240 may be associated with the match camera(s)215.

Similarly, the reference camera IMU 265 includes any combination ofaccelerometer(s), gyroscope(s), and/or compass(es) for generatinginertial tracking data 270 that may be associated with the referencecamera 260 (e.g., where the reference camera IMU 265 is mounted to theuser instrument 250 at a fixed position relative to the reference camera260).

FIG. 2 also illustrates that, in some implementations, the HMD 200 andthe user instrument 250 are configured to share data through a wirelesslink 290. In one example, the user instrument 250 may transmit imagedata for image frames captured by the reference camera 260 to the HMD200 through a wireless channel. It should be noted that the wirelesslink 290 may implement various wireless communication technologies, suchas ultra-wideband, WLAN, infrared communication, Bluetooth, and/orothers.

FIG. 3 illustrates an example of capturing an environment that includesa physical object 305 with a reference camera 260 and a match camera215. In particular, FIG. 3 illustrates the reference camera 260 of theuser instrument 250 capturing a base reference frame 310 at a basereference camera pose 320 and base reference camera timepoint 325. Asillustrated in FIG. 3 , the base reference frame 310 includes arepresentation of the physical object 305. Similarly, FIG. 3 illustratesthe match camera 215 of the HMD 200 capturing a base match frame 315 ata base match camera pose 330 and base match camera timepoint 335. Thebase match frame 315 also includes a representation of the physicalobject 305.

As will be described hereinbelow, the base reference frame 310 and thebase match frame 315 may provide a basis for generating a motion modelfor facilitating mapping of imagery captured by the reference cameraonto imagery captured by the match camera (or vice versa).

FIG. 4 illustrates an example of performing feature matching between thebase reference frame 310 and the base match frame 315. In someimplementations, performing feature matching involves identifyingfeature points and feature descriptors within the base reference frame310 and the base match frame 315. In some instances, a feature point(sometimes referred to as “keypoints,” “points of interest,” or simply“features”) refers to a pixel within an image that comprises richtexture information, such as edges, corners, and/or other readilyidentifiable structures. In some instances, a feature descriptor (alsoreferred to as a “feature vector”) results from extracting imagedata/statistics from a local image/pixel patch around an identifiedfeature point. A feature descriptor may operate as an identifier for thefeature point about which the feature descriptor is centered. Variousapproaches exist for extracting feature descriptors, such as localhistogram approaches, N-jets approaches, and/or others. For example, afeature descriptor may be identified based on a histogram of gradientmagnitudes (e.g., changes in intensity and/or color) and/or orientations(e.g., edge orientations) for pixels within an image patch centered on afeature point.

A system may employ various techniques for identifying feature pointsand/or feature descriptors, such as, by way of non-limiting example,scale-invariant feature transform (SIFT), speeded up robust features(SURF), Canny operator, Kayyali operator, Moravec algorithm, Harris &Stephens/Shi-Tomasi algorithms, Förstner detector, smallest univaluesegment assimilating nucleus (SUSAN) detector, level curve curvatureapproach, DAISY algorithms, and/or others.

FIG. 4 illustrates various feature points associated with the physicalobject 305 as represented in the base reference frame 310 and the basematch frame 315. For example, FIG. 4 illustrates feature points 410A,410B, and 410C on the edges of the depiction of the physical object 305within the base match frame 315. FIG. 4 also depicts feature descriptors415A, 415B, and 415C that are associated, respectively, with featurepoints 410A, 410B, and 410C.

FIG. 4 also illustrates various feature points associated with thedepiction of the physical object 305 within the base reference frame310. For example, FIG. 4 illustrates feature points 420A, 420B, and 420Con the edges of the depiction of the physical object 305 within the basereference frame 310. FIG. 4 also depicts feature descriptors 425A, 425B,and 425C that are associated, respectively, with feature points 420A,420B, and 420C.

As illustrated in FIG. 4 , the feature points 410A, 410B, and 410Cwithin the base match frame 315 correspond to the feature points 420A,420B, and 420C within the base reference frame 310. As used herein,feature points “correspond” to one another when they represent the same3D point within a captured environment. For example, feature point 410Awithin the base match frame 315 and feature point 420A within the basereference frame 310 both represent the same 3D point along the left edgeof the physical object 305.

As used herein, a “feature match” comprises a feature point in one image(e.g., the base reference frame 310) and a corresponding feature pointin another image (e.g., the base match frame 315). In some instances, asystem identifies feature matches by comparing feature descriptors ofthe features identified in the images. A system may employ varioustechniques to identify feature matches between the features of the basematch frame 315 and the base reference frame 310, such as a brute-forcematcher, a fast library for approximate nearest neighbors (FLANN)matcher, and/or others. FIG. 4 illustrates three feature matches (e.g.,feature points 410A and 420A, feature points 410B and 420B, and featurepoints 410C and 420C), and the ellipses 450 and 460 indicate that asystem may identify any number of feature matches within the basereference frame 310 and the base match frame 315 (e.g., in someinstances, on the order of 100 matches).

In some instances, a system utilizes feature matches identified withinthe base match frame 315 and the base reference frame 310 to determine amotion model for facilitating mapping of imagery captured by thereference camera onto imagery captured by the match camera (or viceversa). FIG. 5 illustrates an example of unprojecting the featurematches identified within the base match frame 315 and the basereference frame 310.

Initially, FIG. 5 illustrates a colocation point 500, which illustratesan assumption that the reference camera 260 and the match camera 215were at a same location while capturing the base match frame 315 and thebase reference frame 310 (e.g., the separation distance between thereference camera 260 and the match camera 215 is set to zero). However,while the colocation point 500 illustrates an assumption that thereference camera 260 and the match camera 215 shared a same locationwhile capturing images, the colocation point 500 does not constrain theorientation of the reference camera 260 and the match camera 215. Forexample, FIG. 5 depicts the reference camera 260 at the colocation point500 with its orientation determined by the base reference camera pose320, and FIG. 5 also depicts the match camera 215 at the colocationpoint 500 with its orientation determined by the base match camera pose330.

FIG. 5 illustrates generating unprojected feature points 520A, 520B, and520C by performing unprojection 530 on the feature points 420A, 420B,and 420C of the base reference frame 310. FIG. 5 also illustratesgenerating unprojected feature points 510A, 510B, and 510C by performingunprojection 540 on the feature points 410A, 410B, and 410C of the basematch frame 315. The unprojected feature points are 3D points that aregenerated based on the 2D feature points.

In some instances, generating 3D unprojected feature point (e.g.,510A-510C, 520A-520C) by performing unprojection (e.g., unprojection540, 530) on a 2D feature point (e.g., feature points 410A-410C,420A-420C) may be conceptualized as extending a ray from a camera centeror optical center using a pixel location of the feature point to beunprojected. In one example for generating unprojected feature point520A, using pinhole camera terminology for illustrative purposes, asystem may define the colocation point 500 as an optical center orcamera center of the reference camera 260 while the reference cameracaptured the base reference frame 310 at the base reference camera pose320. Continuing with the example, the system may then cast a ray fromthe colocation point 500 using the 2D pixel coordinates of feature point420A as feature point 420A lies on a front image plane positioned aboutthe colocation point 500. The front image plane may be positioned aboutthe colocation point 500 according to the base reference camera pose 320(e.g., wherein the base reference camera pose 320 provides theorientation of the front image plane with respect to the colocationpoint 500). The system may extend the ray to a particular depth value toprovide the 3D unprojected feature point 520A. A system may use anydepth value for the unprojection 530, such as a uniform depth value forgenerating all unprojected feature points.

The ellipses 550 and 560 of FIG. 5 indicate that a system may performunprojection 530, 540 to generate any number of 3D unprojected featurepoints based on the 2D feature points of the base reference frame 310and the base match frame 315. As noted hereinabove, at least some of thefeature points of the base reference frame 310 (e.g., feature points420A-420C) may correspond to feature points of the base match frame 315(e.g., feature points 410A-410C), providing feature matches between thebase reference frame 310 and the base match frame 315. Therefore, FIG. 5illustrates an instance in which performing unprojection 530, 540 on thefeature points of the base reference frame 310 and the base match frame315 provides unprojected feature matches 570 of correspondingunprojected feature points. For example, unprojected feature points 510Aand 520A may correspond to one another, being generated by unprojectingfeature points 410A and 420A, respectively.

FIG. 6 illustrates an example of identifying a base matrix 610 using theunprojected feature matches 570. In some implementations, a base matrix610 represents a 3D rotation matrix, or a 3×3 matrix that describesrotation about 3 perpendicular axes to rotate a set of points from onecoordinate system into a different coordinate system. In some instances,as indicated by FIG. 6 , the base matrix 610 is a 3D rotation matrixthat facilitates rotation of the unprojected feature points 520A-520Cinto the coordinate system of the corresponding unprojected featurepoints 510A-510C (the corresponding unprojected feature points 510A-510Chaving been unprojected using the base match camera pose 330). Putdifferently, the base matrix 610 is a 3D rotation matrix that rotates aset of unprojected feature points (e.g., 520A-520C) onto a set ofcorresponding unprojected feature points (e.g., 510A-510C).

By way of illustration, FIG. 6 depicts the reference camera 260positioned at the colocation point 500 with an orientation thatcorresponds to the base reference camera pose 320. It should be notedthat the base reference camera pose 320 is the pose from which thereference camera 260 captured the base reference frame 310 that providedthe 2D feature points 420A-420C for generating unprojected featurepoints 520A-520C. FIG. 6 illustrates that the base matrix 610 may bedescribed as a 3D rotation matrix that facilitates rotation of thereference camera 260 to correspond to the base match camera pose 330,which is the pose from which the match camera 215 captured the basematch frame 315 that provided the 2D feature points 410A-410C forgenerating unprojected feature points 510A-510C.

A system may identify a base matrix 610 using unprojected featurematches 570 in a variety of ways, such as by minimizing a cost function(e.g., according to the Wahba method), utilizing machine learningtechniques, and/or other approaches.

Using a base matrix 610, a system may map pixels from the base referenceframe 310 onto corresponding pixels of the base match frame 315, asdiscussed in more detail with reference to FIGS. 7-10 . FIG. 7illustrates generating 3D points 740 by performing unprojection 730 on aset of pixels of the base reference frame 310 captured by the referencecamera 260 at the base reference camera pose 320. Unprojection 730corresponds to unprojection 530, 540 described hereinabove withreference to FIG. 5 . In some instances, a system performs unprojection730 on pixels of the base reference frame 310 using the intrinsic matrix(e.g., intrinsic calibration) of the reference camera 260 and a uniformdepth value. Ellipsis 750 indicates that a system may performunprojection 730 on any number of pixels of the base reference frame 310to generate the 3D points 740.

FIG. 8 illustrates an example of generating modified 3D points 860 byapplying the base matrix 610 to the 3D points 740. In some instances,applying the base matrix 610 to the 3D points 740 rotates the 3D points740 into a different coordinate system, thereby providing modified 3Dpoints 860. By way of illustration, applying the base matrix 610 to the3D points 740 may be thought of as rotating the 3D points 740 into acoordinate system that would exist for 3D points unprojected using thebase match frame 315 and the base match camera pose 330. Ellipsis 850indicates that a system may apply the base matrix 610 to any number ofthe 3D points 740 to generate the modified 3D points 860.

FIG. 9 illustrates an example of performing projection 930 on themodified 3D points 860. In some instances, projection 930 is anoperation that is inverse to the unprojection operations describedherein (e.g., unprojection 530, 540, 730). Projection 930 may beconceptualized as extending a ray from a 3D point (e.g., one of themodified 3D points 860) toward a camera center or optical center. Forinstance, again using pinhole camera terminology for illustrativepurposes, a system may define the colocation point 500 as an opticalcenter or camera center of the match camera 215 while the match cameracaptured the base match frame 315 at the base match camera pose 330. Asystem may then cast a ray from a 3D point (e.g., one of the modified 3Dpoints 860) toward the colocation point 500, and the ray may identify a2D pixel as the pixel lies on a front image plane positioned about thecolocation point. The front image plane may be positioned about thecolocation point according to an intrinsic matrix (e.g., intrinsiccalibration) of the match camera 215 and according to the base matchcamera pose 330 (e.g., wherein the base match camera pose 330 providesthe orientation of the front image plane with respect to the colocationpoint 500). The system may then associate the projected 3D point withthe 2D pixel coordinates identified according to the projection 930 (andthereby associate 2D pixel coordinates with the pixels of the basereference frame 310 that were originally unprojected according tounprojection 730). The ellipsis 950 indicates that a system may performprojection 930 on any number of the modified 3D points 860 to associatethe modified 3D points with 2D pixel coordinates. The 2D pixelcoordinates may identify a set of corresponding pixels in the base matchframe 315 to which a system may map of pixels from the base referenceframe 310.

In some instances, a system uses the 2D pixel coordinates associatedwith the modified 3D points 860 to identify a set of correspondingpixels in the base match frame 315 to which the system may map the setof pixels of the base reference frame 310 that were originallyunprojected according to unprojection 730. FIG. 10 illustrates anexample of a composite frame 1010 that includes base reference framepixels 1030 mapped to corresponding base match frame pixels 1040 in anoverlap region 1020 of the composite frame 1010. For example, in someinstances, the system applies texture information from the basereference frame pixels 1030 to the corresponding base match frame pixels1040 that were identified according to the projections 930 of themodified 3D points 860.

Accordingly, FIGS. 7-9 illustrate facilitating image alignment of a basereference frame and a base match frame using a 3D rotational matrix(i.e., base matrix 610), which is a simplification made possible by thecolocation assumption described hereinabove and exemplified bycolocation point 500 from FIGS. 5-9 . The colocation assumption mayreduce the computation burden associated with continuously mappingimagery from one camera onto imagery of another camera. The colocationassumption ignores real-world parallax that may exist between theperspective of the match camera and the perspective of the referencecamera. However, in some implementations, the effects of parallax areonly observable when the cameras capture objects that are relativelyclose to the cameras, and/or when the separation distance between thecameras is relatively large.

Although the foregoing description focuses, in some respects, onimplementations that use the colocation assumption, those skilled in theart will recognize, in view of the present disclosure, that at leastsome of the principles described herein are applicable inimplementations that omit the colocation assumption.

In some instances, the operations of generating a set of 3D points 740by unprojecting a set of pixels of the base reference frame 310,generating modified 3D points 860 by applying the base matrix 610 to the3D points 740, and projecting the modified 3D points 860 as describedhereinabove with reference to FIGS. 7-9 map the set of pixels of thebase reference frame 310 onto a set of corresponding pixels in the basematch frame 315. These operations may be thought of as a model thatcaptures the relative positioning of the reference camera 260 and thematch camera 215 to enable mapping of pixels from a base reference frameto a base match frame. The model may be represented by the followingequation:

f(p)=K _(match) *R _(ref→match) *K _(ref) ⁻¹  (1)

Where f(p) is a function that maps pixels p of the reference frame topixels p′ of the match frame. K_(ref) represents the intrinsic matrix ofthe reference camera, and K_(match) represents the intrinsic matrix ofthe match camera. R_(ref→match) represents an alignment matrix, whichmay be implemented as the base matrix described hereinabove (or othertransformations, such as homography, similarity transforms, affinemotion models, etc.). In some instances, Equation 1 representsunprojection of a pixel p of the reference frame using the intrinsicmatrix of the reference camera, K_(ref), rotating the unprojected pointusing the base matrix, R_(ref→match), and projecting the rotated pointonto the match frame using the intrinsic matrix of the match camera,K_(match).

Attention is now directed to FIG. 11 , which illustrates an example ofthe reference camera 260 of the user instrument 250 and the match cameraof the HMD 200 continuously capturing the environment that includes thephysical object 305 as the poses of the user instrument 250 and the HMD200 change over time. Specifically, FIG. 11 illustrates the referencecamera 260 of the user instrument 250 capturing an updated referenceframe 1110 at an updated reference camera pose 1120 and at an updatedreference camera timepoint 1125. The updated reference camera pose 1120is different than the base reference camera pose 320, and the updatedreference camera timepoint 1125 is subsequent to the base referencecamera timepoint 325. Similarly, FIG. 11 illustrates the match camera215 of the HMD 200 capturing an updated match frame 1115 at an updatedmatch camera pose 1130 and at an updated match camera timepoint 1135.The updated match camera pose 1130 is different than the base matchcamera pose 330, and the updated match camera timepoint 1135 issubsequent to the base match camera timepoint 335.

In some instances, the difference between the base reference camera pose320 and the updated reference camera pose 1120 is captured by thereference camera IMU 265 in the form of a reference cameratransformation matrix 1170. In some implementations, the referencecamera transformation matrix 1170 is a 3D rotational matrix identifiedbased on inertial tracking data 270 obtained by the reference camera IMU265. Similarly, the difference between the base match camera pose 330and the updated match camera pose 1130 is captured by the match cameraIMU 235 in the form of a match camera transformation matrix 1140. Insome implementations, the match camera transformation matrix 1140 isalso a 3D rotational matrix identified based on inertial tracking data240 obtained by the match camera IMU 235.

As is evident in FIG. 11 , because of the pose change of the referencecamera 260, the position of the depiction of the physical object 305within the updated reference frame 1110 is different when compared withthe position of the depiction of the physical object 305 within the basereference frame 310. Similarly, because of the pose change of the matchcamera 215, the position of the depiction of the physical object 305within the updated match frame 1115 is different when compared with theposition of the depiction of the physical object 305 within the basematch frame 315. Thus, the base matrix 610, if used alone, may fail toaccurately map a set of pixels of the updated reference frame 1110 to aset of corresponding pixels of the updated match frame 1115 because ofthe pose differences of the cameras between the base timepoint(s) (e.g.,the base reference camera timepoint 325 and the base match cameratimepoint 335) and the updated timepoint(s) (e.g., the updated referencecamera timepoint 1125 and the updated match camera timepoint 1135).

However, in some instances, utilizing the reference cameratransformation matrix 1170 and/or the match camera transformation matrix1140 in combination with the base matrix 610 may enable the system tomap a set of pixels of the updated reference frame 1110 to a set ofcorresponding pixels of the updated match frame 1115.

For example, FIG. 12 illustrates an example of generating 3D points 1240by performing unprojection 1230 on a set of pixels of the updatedreference frame 1110 captured by the reference camera 260 at the updatedreference camera pose 1120. In some instances, unprojection 1230corresponds to unprojection 730 described hereinabove with reference toFIG. 7 . For example, in some instances, a system performs unprojectionon the set of pixels of the updated reference frame 1110 from thecolocation point 1200 using the updated reference camera pose 1120 ofthe reference camera 260. The ellipsis 1250 indicates that a system mayperform unprojection 1230 on any number of pixels of the updatedreference frame 1110 to generate the 3D points 1240.

FIG. 13 illustrates an example of generating modified 3D points 1360 byapplying the reference camera transformation matrix 1170 to the 3Dpoints 1240. In some instances, applying the reference cameratransformation matrix 1170 rotates the 3D points 1240 into thecoordinate system that existed for the 3D points 740 generated byperforming unprojection 730 using the base reference frame 310 and thebase reference camera pose 320. For illustrative purposes, FIG. 13depicts the reference camera 260 positioned at the colocation point 1200according to the base reference camera pose 320. Accordingly, applyingthe reference camera transformation matrix 1170 to the 3D points 1240 togenerate the modified 3D points 1360 may be thought of as restoring thepose of the reference camera 260 that existed while the reference camera260 captured the base reference frame 310 (e.g., base reference camerapose 320). The ellipsis 1350 indicates that a system may apply thereference camera transformation matrix 1170 to any number of 3D points1240 to generate the modified 3D points 1360.

FIG. 14 illustrates an example of generating modified 3D points 1460 byapplying the base matrix 610 to the modified 3D points 1360. In someinstances, related to applying the base matrix 610 to the 3D points 740described hereinabove with reference to FIG. 7 , applying the basematrix 610 to the modified 3D points 1360 rotates the modified 3D pointsinto a coordinate system that would exist for 3D points unprojectedusing the base match frame 315 and the base match camera pose 330 (whichmay be the same coordinate system that existed for the modified 3Dpoints 860 described hereinabove with reference to FIG. 8 ). Forillustrative purposes, FIG. 14 depicts the match camera 260 positionedat the colocation point 1200 according to the base match camera pose330. The ellipsis 1450 indicates that a system may apply the base matrix610 to any number of modified 3D points 1360 to generate the modified 3Dpoints 1460.

FIG. 15 illustrates an example of generating modified 3D points 1560 byapplying the match camera transformation matrix 1140 to the modified 3Dpoints 1460. In some instances, applying the match camera transformationmatrix 1140 rotates the modified 3D points 1460 into a coordinate systemthat would exist for 3D points unprojected using the updated match frame1115 and the updated match camera pose 1130. The ellipsis 1550 indicatesthat a system may apply the match camera transformation matrix 1140 toany number of modified 3D points 1460 to generate the modified 3D points1560.

FIG. 16 illustrates an example of performing projection 1630 on themodified 3D points 1560. In some instances, projection 1630 correspondsto projection 930 described hereinabove with reference to FIG. 8 . Forexample, in some instances, a system performs projection 1630 on themodified 3D points 1560 toward the colocation point 1200 using theupdated match camera pose 1130 of the match camera 215 to associate 2Dpixel coordinates with the projected modified 3D points 1560 (andthereby associated 2D pixel coordinates with the pixels of the updatedreference frame 1110 that were originally unprojected according tounprojection 1230).

As before, in some implementations, a system uses the 2D pixelcoordinates associated with the modified 3D points 1560 to identify aset of corresponding pixels in the updated match frame 1115 to which thesystem may map the set of pixels of the updated reference frame 1110that was originally unprojected according to unprojection 1230. FIG. 17illustrates an example of a composite frame 1710 that includes updatedreference frame pixels 1730 mapped to updated corresponding match framepixels 1740 in an overlap region 1720 of the composite frame 1710. Forexample, in some instances, the system applies texture information fromthe updated reference frame pixels 1730 to the updated correspondingmatch frame pixels 1740 that were identified according to theprojections 1630 of the modified 3D points 1560.

Accordingly, FIGS. 12-16 illustrate facilitating image alignment of anupdated reference frame and an updated match frame using a combinationof 3D rotational matrices (in particular, the base matrix 610, thereference camera transformation matrix 1170, and the match cameratransformation matrix 1140). The operations described hereinabove withreference to FIGS. 12-16 may be thought of as a motion model that mapsthe relative positioning of the reference camera 260 and the matchcamera 215 over time to enable continuous mapping of pixels from updatedreference frames to updated match frames. The motion model may berepresented by the following equation:

f(p)=K _(match) *P _(match_cur) *P _(match_base) ⁻¹ *R _(ref→match) *P_(ref_base) *P _(ref_cur) ⁻¹ *K _(ref) ⁻¹  (2)

Where f(p) is a function that maps pixels p of the reference frame topixels p′ of the match frame. As before, K_(ref) represents theintrinsic matrix of the reference camera, and K_(match) represents theintrinsic matrix of the match camera. R_(ref→match) represents analignment matrix, which may be implemented as the base matrix describedhereinabove. P_(ref_base) and P_(match_base) refer to the base referencecamera pose 320 and the base match camera pose 330, respectively, andP_(ref_cur) and P_(match_cur) refer to the updated reference camera pose1120 and the updated match camera pose 1130, respectively. Accordingly,in some implementations, the combination of P_(ref_base)*P_(ref_cur) ⁻¹refers to the reference camera transformation matrix 1170 describedhereinabove. Furthermore, in some instances, the combination ofP_(match_cur)*P_(match_base) ⁻¹ refers to the match cameratransformation matrix 1140 described hereinabove. Accordingly, in someimplementations, Equation 2 may be thought of as a motion modelconfigured to facilitate continuous mapping of sets of pixels of updatedreference frames captured by the reference camera to corresponding setsof pixels of updated match frames captured by the match camera, with themotion model being based on the base matrix, the reference cameratransformation matrix, and the match camera transformation matrix.

However, inertial tracking data obtained by IMUs (e.g., inertialtracking data 240 obtained by the match camera IMU 235 and/or inertialtracking data 270 obtained by the reference camera IMU 265) may be proneto drift, which refers to accumulated errors brought about bycontinually integrating acceleration with respect to time. Accordingly,the accuracy of the motion model represented by Equation 2 and describedhereinabove with reference to FIGS. 12-16 may degrade as poses continueto update from the time that the original base matrix 610 was computed.

Accordingly, in some implementations, a motion model utilizes analignment matrix that is generated based on a previously computed basematrix and/or a current updated matrix (e.g., by fusing the twotogether). Similar to the base matrix 610 described hereinabove, in someinstances, an updated matrix is also generated/identified using visualcorrespondences between frames captured by the reference camera 260 andthe match camera 215. In addition to combatting drift, generating amotion model using matrices generated from visual correspondences offrame pairs captured at different timepoints may also ameliorate theeffects that changes in the relative positioning of the cameras may haveon composite images generated using the motion model.

FIG. 18 illustrates an example of performing feature matching betweenthe updated reference frame 1110 (captured by the reference camera 260at updated reference camera pose 1120 and timepoint 1125, see FIG. 11 )and the updated match frame 1115 (captured by the match camera 215 atupdated match camera pose 1130 and timepoint 1135, see FIG. 11 ).Similar to FIG. 4 described hereinabove, FIG. 18 illustrates variousfeature points associated with the physical object 305 as represented inthe updated reference frame 1110 and the updated match frame 1115. Forexample, FIG. 18 illustrates feature points 1810A, 1810B, and 1810C onthe edges of the depiction of the physical object 305 within the updatedmatch frame 1115. FIG. 18 also depicts feature descriptors 1815A, 1815B,and 1815C that are associated, respectively, with feature points 1810A,1810B, and 1810C.

FIG. 18 also illustrates various feature points associated with thedepiction of the physical object 305 within the updated reference frame1110. For example, FIG. 18 illustrates feature points 1820A, 1820B, and1820C on the edges of the depiction of the physical object 305 withinthe updated reference frame 1110. FIG. 18 also depicts featuredescriptors 1825A, 1825B, and 1825C that are associated, respectively,with feature points 1820A, 1820B, and 1820C.

As illustrated in FIG. 18 , the feature points 1810A, 1810B, and 1810Cwithin the updated match frame 1115 correspond to the feature points1820A, 1820B, and 1820C within the updated reference frame 1110. Forexample, feature point 1810A within the updated match frame 1115 andfeature point 1820A within the updated reference frame 1110 bothrepresent the same 3D point along the left edge of the physical object305. Each pair of feature points that correspond to one another formfeature matches. The ellipses 1850 and 1860 indicate that a system mayidentify any number of feature matches within the updated referenceframe 1110 and the updated match frame 1115.

FIG. 19 illustrates an example of unprojecting the feature matchesidentified within the updated reference frame 1110 and the updated matchframe 1115. In some instances, using the colocation point 1900 and theupdated reference camera pose 1120 of the reference camera 260, a systemperforms unprojection 1930 on the various 2D feature points identifiedin the updated reference frame 1110 (e.g., feature points 1820A-1820C)to generate 3D unprojected feature points (e.g., unprojected featurepoints 1920A-1920C). Similarly, in some instances, using the colocationpoint 1900 and the updated match camera pose 1130 of the match camera215, a system performs unprojection 1940 on the various 2D featurepoints identified in the updated match frame 1115 (e.g., feature points1810A-1810C) to generate 3D unprojected feature points (e.g.,unprojected feature points 1910A-1910C). The unprojections 1930, 1940may correspond, in at least some respects, to the unprojections 530, 540described hereinabove with reference to FIG. 5 , and performing theunprojections 1930, 1940 may provide 3D unprojected feature matches1970. The ellipses 1950 and 1960 indicate that a system may perform anynumber of unprojections 1930, 1940 to generate any number of 3Dunprojected feature matches 1970.

FIG. 20 illustrates an example of identifying an updated matrix 2010using the unprojected feature matches 1970. As with the base matrix 610described hereinabove, in some instances, an updated matrix 2010represents a 3D rotation matrix that facilitates rotation of theunprojected feature points 1920A-1920C into the coordinate system of thecorresponding unprojected feature points 1910A-1910C (the correspondingunprojected feature points 1910A-1910C having been unprojected using theupdated match camera pose 1130). Furthermore, as with the base matrix610, a system may identify an updated matrix 2010 using unprojectedfeature matches 1970 in a variety of ways, such as by minimizing a costfunction (e.g., according to the Wahba method), utilizing machinelearning techniques, and/or other approaches.

Although, in some implementations, a system may utilize the updatedmatrix 2010 to facilitate mapping of pixels of the updated referenceframe 1110 to pixels of the updated match frame 1115 (e.g., by utilizingthe updated matrix 2010 as R_(ref→match) in Equation 1), it should benoted that feature point identification is a noisy process. For example,the pixel coordinates of features points that describe 3D points of acaptured environment may shift from expected positions acrossconsecutively captured frames. Such occurrences may cause observablespatial flickers in overlap regions of composite frames. Furthermore, insome instances, at least some feature points that arepresent/identifiable in one image may not be present/identifiable inanother image, even where the different images are captured using thesame camera modality (e.g., because of occlusions).

Accordingly, in some implementations, a system aligns the updated matrix2010 with a previously computed matrix (e.g., base matrix 610) togenerate an alignment matrix by fusion. Utilizing such an alignmentmatrix in a motion model for mapping pixels from reference frames tomatch frames may have the effect of smoothing out noise that mayotherwise affect composite frames generated using the motion model.

FIGS. 21 and 22 illustrate an example of generating an aligned updatedmatrix 2210 by modifying the updated matrix 2010 using inertial trackingdata associated with the reference camera 260 and the match camera 215.As noted hereinabove, in some instances, the inertial tracking data 270obtained by the reference camera IMU 265 tracks the pose changes of thereference camera 260 from the base reference camera timepoint 325 to theupdated reference camera timepoint 1125 (e.g., by identifying the basereference camera pose 320 and the updated reference camera pose 1120).Similarly, the inertial tracking data 240 obtained by the match cameraIMU 235 tracks the pose changes of the match camera 215 from the basematch camera timepoint 335 to the updated match camera timepoint 1135(e.g., by identifying the base match camera pose 330 and the updatedmatch camera pose 1130).

In some instances, a system utilizes the inertial tracking datadescribed above to modify the updated matrix 2010 to bring the updatedmatrix 2010 into the same reference/coordinate system that exists forthe base matrix 610. FIG. 21 illustrates a conceptual representation ofa system modifying the 3D rotation described by the updated matrix 2010by identifying a reference camera rotational matrix 2130 and a matchcamera rotational matrix 2150 based on the inertial tracking data thatdescribes the motion of the reference camera 260 and the match camera215 between capturing the base frames (e.g., the base reference frame310 and the base match frame 315) and the updated frames (e.g., theupdated reference frame 1110 and the updated match frame 1115). Thesystem modifies the updated matrix 2010 with the reference camerarotational matrix 2130 and the match camera rotational matrix 2150 togenerate the aligned updated matrix 2210, as shown in FIG. 22 .Generating the aligned updated matrix 2210 by using the reference camerarotational matrix 2130 and the match camera rotational matrix 2150 maybe conceptualized as undoing the motion that occurred between when thebase fames were captured (e.g., the base reference frame 310 and thebase match frame 315) and when the updated frames were captured (e.g.,the updated reference frame 1110 and the updated match frame 1115).Generating an aligned updated matrix 2210 may be represented by thefollowing equation:

R′ _(ref→match) =P _(match_base) *P _(match_cur) ⁻¹ *R′_(ref_cur→match_cur) *P _(ref_cur) *P _(ref_base) ⁻¹  (3)

Where R′_(ref→match) may represent the aligned updated matrix 2210. Asbefore, R_(ref→match) may represent the base matrix 610 describedhereinabove. Furthermore, as before, P_(ref_base) and P_(match_base)refer to the base reference camera pose 320 and the base match camerapose 330, respectively, and P_(ref_cur) and P_(match_cur) refer to theupdated reference camera pose 1120 and the updated match camera pose1130, respectively. In some implementations, the combination ofP_(ref_cur)*P_(ref_base) ⁻¹ refers to the reference camera rotationalmatrix 2130 described hereinabove. Furthermore, in some instances, thecombination of P_(match_base)*P_(match_cur) ⁻¹ refers to the matchcamera rotational matrix 2150 described hereinabove.

FIG. 22 also depicts the base matrix 610 and illustrates that, in someinstances, although the aligned updated matrix 2210 and the base matrix610 are computed to share the same reference/coordinate system,differences exist between the base matrix 610 and the aligned updatedmatrix 2210 (e.g., because of IMU drift, noise in performing featurematching, changes in relative positioning of the reference camera 260and the match camera 215, etc.). Accordingly, in some implementations, asystem fuses the base matrix 610 with the aligned updated matrix 2210 togenerate an alignment matrix for a motion model to facilitate continuousimage alignment of separate camera imagery.

FIG. 23A illustrates an example of generating an alignment matrix 2315using the base matrix 610 and the aligned updated matrix 2210. FIG. 23Aillustrates that, in some instances, a system uses the alignment updatedmatrix 2210 and the base matrix 610 as inputs for a fuser 2300. In theimplementation depicted in FIG. 23A, the fuser 2300 implements aninterpolation function 2305, which, in essence, blends or combines thealigned updated matrix 2210 with the base matrix 610 to generate thealignment matrix 2315. The alignment matrix 2315 is, in some instances,also a 3D rotation matrix.

FIG. 23A also illustrates that, in some instances, the interpolationfunction 2305 utilizes a smoothness term 2310 (or smoothness function).In some implementations, the smoothness term determines how to weightthe aligned updated matrix 2210 and the base matrix 610 for generatingthe alignment matrix 2315. By way of example, in some implementations,when the smoothness term 2310 has a value that approaches a maximumvalue (e.g., a value of 1), the interpolation function 2305 increasinglyignores the aligned updated matrix 2210, giving more weight to the basematrix 610 for generating the alignment matrix 2315, which may bebeneficial when the expected accuracy of the base matrix 610 is high.Furthermore, in some implementations, when the smoothness term 2310 hasa value that approaches a minimum value (e.g., a value of 0), theinterpolation function 2305 increasingly ignores the base matrix 610,giving more weight to the aligned updated matrix 2210 for generating thealignment matrix 2315, which may be beneficial when the expectedaccuracy of the base matrix 610 is low.

In some instances, the smoothness term 2310 is set to a constant valuebetween (e.g., a value between 0 and 1, such as a value of 0.7), whereasin other instances, the smoothness term 2310 is intelligentlydetermined/updated based on various factors (e.g., an expected accuracyof the base matrix). Additional details concerning intelligentlydetermining a smoothness term 2310 will be provided hereinafter.

The ellipsis 2330 indicates that, in some instances, a fuser 2300incorporates additional and/or alternative components.

Generating an alignment matrix 2315 using an aligned updated matrix 2210and a base matrix 610 (or a previous alignment matrix) may berepresented by the following equation:

R _(ref→match) interpolate(R′ _(ref→match) ,R _(ref→match),smoothness())  (4)

Where R_(ref→match) on the left side of Equation 4 may represent thealignment matrix 2315 described hereinabove. The function interpolate( )may represent the interpolation function 2305 of the fuser 2300described hereinabove, and the function smoothness( ) may represent thesmoothness term 2310 described hereinabove. R_(ref→match) may refer tothe aligned updated matrix 2210, and R_(ref→match) on the right side ofthe equation may refer to the base matrix 610, or, in some instances,may refer to a previously computed alignment matrix. For example, FIG.23B illustrates a fuser 2300 receiving the alignment matrix 2315generated according to FIG. 23A and a subsequent aligned updated matrix2320 (e.g., an aligned updated matrix generated subsequent to generatingthe alignment matrix 2315 from FIG. 23A) as inputs for generating asubsequent alignment matrix 2325. Thus, in some instances, a systemutilizes an identified alignment matrix in conjunction with a subsequentupdated matrix in order to identify a subsequent alignment matrix.Accordingly, generating alignment matrices may be thought of as arunning average that incorporates past alignment matrices to generatenew alignment matrices.

In some instances, a system utilizes the alignment matrix 2315 as partof the motion model for mapping a set of pixels from a reference frameto a corresponding set of pixels of a match frame (e.g., by utilizingR_(ref→match) on the left side of Equation 4 as R_(ref→match) inEquation 2). For example, FIG. 24 illustrates an example of a motionmodel 2430 configured to map a reference frame 2410 onto a match frame2420. The motion model 2430 of FIG. 24 includes a reference cameraintrinsic matrix 2435 and a match camera intrinsic matrix 2455 (e.g., tofacilitate the unprojection and/or projection operations describedhereinabove, such as those referred to with reference to FIGS. 5, 7, 9,12, 16, and 19 and/or with reference to Equations 1 and 2). The motionmodel 2430 of FIG. 24 also includes a match camera transformation matrix2440 and a reference camera transformation matrix 2445 (e.g., forapplication to 3D points and/or to modify/align updated matrices, asdepicted in FIGS. 13, 15, and 21 and/or in Equations 2 and 3). Themotion model 2430 also includes an alignment matrix 2450, which maycorrespond to a base matrix 610, an aligned updated matrix 2210, analignment matrix 2315, and/or a subsequent alignment matrix 2325described hereinabove. The ellipsis 2465 indicates that a motion model2430 may comprise any number of components, including or different thanthose depicted in FIG. 24 .

In some instances, a system utilizes the motion model 2430 to map pixelsof a reference frame 2410 to pixels of a match frame 2420 in order togenerate a composite image 2460. For example, in some implementations, asystem generates 3D points by unprojecting pixels of the reference frame2410 using the reference camera intrinsic matrix 2435; generatesmodified 3D points by applying the reference camera transformationmatrix 2445, the alignment matrix 2450, and the match cameratransformation matrix 2440 to the 3D points; and projects the modified3D points to generate a composite image 2460 that overlays the pixels ofthe reference frame 2410 onto corresponding pixels of the match frame2420.

Example Method(s) for Continuous Image Alignment of Separate Cameras

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed.

FIG. 25 illustrates an example flow diagram 2500 depicting actsassociated with generating a motion model configured to facilitatemapping of a set of pixels of a reference frame captured by a referencecamera to a corresponding set of pixels of a match frame captured by amatch camera. The discussion of the various acts represented in flowdiagram 2500 includes references to various hardware componentsdescribed in more detail with reference to FIGS. 1, 2, and 34 .

Act 2502 of flow diagram 2500 includes obtaining an updateable basematrix. Act 2502 is performed, in some instances, using one or moreprocessors 3405 of a computer system 3400 (e.g., an HMD 200 and/or auser instrument 250) and using images captured by a reference camera 260and a match camera 215. For example, in some instances, the updateablebase matrix is a 3D rotation matrix obtained using based on visualcorrespondences between a base reference frame captured by the referencecamera at a base reference camera pose and a base match frame capturedby the match camera at a base match camera pose.

In some implementations, a computer system obtains an updateable basematrix by identifying a set of base feature matches by performingfeature matching between the base reference frame and the base matchframe. The computer system then generates a set of unprojected basefeature matches by unprojecting the base feature matches into 3D space.The computer system then computes the updateable base matrix based onthe set of unprojected base feature matches by minimizing a costfunction (e.g., according to the Wahba method).

Act 2504 of flow diagram 2500 includes computing an updated matrix. Act2504 is performed, in some instances, using one or more processors 3405of a computer system 3400 (e.g., an HMD 200 and/or a user instrument250) and using images captured by a reference camera 260 and a matchcamera 215. In some instances, a computer system computes an updatedmatrix using visual correspondences between an updated reference framecaptured by the reference camera at an updated reference camera pose andan updated match frame captured by the match camera at an updated matchcamera pose for generating an updated matrix.

In some implementations, a computer system computes an updated matrix byidentifying a set of updated feature matches by performing featurematching between the updated reference frame and the updated matchframe. The computer system also generates a set of unprojected updatedfeature matches by unprojecting the updated feature matches into 3Dspace, and the computer system computes the updated matrix based on theset of unprojected updated feature matches by minimizing a costfunction. In some implementations, the computer system computes theupdated matrix using the Wahba method.

In response to successfully computing an updated matrix according to act2504, act 2506 of flow diagram 2500 includes generating an alignedmatrix, and act 2508 of flow diagram 2500 includes updating theupdateable base matrix. Acts 2506 and 2508 are performed, in someinstances, using one or more processors 3405 of a computer system 3400(e.g., an HMD 200 and/or a user instrument 250). In some instances, acomputer system generates an aligned updated matrix (according to act2506) using a base reference camera pose, a base match camera pose, anupdated reference camera pose, and an updated match camera pose, whichmay be poses associated with the updateable base matrix obtainedaccording to act 2502. Furthermore, in some instances, a computer systemupdates the updateable base matrix (according to act 2508) by using thealigned updated matrix and the updateable base matrix (as previouslyobtained according to act 2502) as inputs for updating the updateablebase matrix. The computer system may then proceed to perform act 2510,which includes generating a motion model using the updateable basematrix.

It should be noted that, in some instances, a computer system fails tocompute an updated matrix. For example, in some instances, the matchcamera and the reference camera are not directed toward a common portionof a captured environment which may cause a system to fail to identifyfeature correspondences between a reference frame and a match frame. Acomputer system may fail to identify feature points for other reasons aswell, such as occlusions, differences in camera modalities, etc. Failureto identify feature points within the match frame and/or the referenceframe may cause the system to fail to compute an updated matrix.

In response to failing to compute an updated matrix according to act2504, in some implementations, a computer system refrains from updatingthe updateable base matrix obtained according to act 2502. Instead, thecomputer system proceeds to act 2510 with the updateable base matrixobtained according to act 2502 (without updating the updateable basematrix, in contrast with act 2508 performed in response to successfullycomputing an updated matrix according to act 2504).

Act 2510 of flow diagram 2500 includes generating a motion model usingthe updateable base matrix (whether the updateable base matrix wasupdated according to act 2508 or not). Act 2510 is performed, in someinstances, using one or more processors 3405 of a computer system 3400(e.g., an HMD 200 and/or a user instrument 250). In some instances, acomputer system the motion model includes the updateable base matrix, areference camera transformation matrix generated based on inertialtracking data associated with the reference camera, and a match cameratransformation matrix generated based on inertial tracking dataassociated with the match camera.

Arrow 2512 of flow diagram 2500 indicates that, in some instances, anupdateable base matrix obtained according to act 2502 is updateable basematrix that was used to generate the motion model according to act 2510,whether the updateable base matrix was updated according to act 2508 ornot. In this sense, an updateable base matrix may be regarded as arunning average.

FIG. 26 illustrates an example flow diagram 2600 depicting actsassociated with facilitating continuous image alignment of two cameras.The discussion of the various acts represented in flow diagram 2600includes references to various hardware components described in moredetail with reference to FIGS. 1, 2, and 34 .

Act 2602 of flow diagram 2600 includes identifying a reference cameratransformation matrix. Act 2602 of flow diagram 2600 is performed, insome instances, using one or more processors 3405 of a computer system3400 (e.g., an HMD 200 and/or a user instrument 250). In someimplementations, the reference camera transformation matrix is a 3Drotational matrix (e.g., identified based on inertial tracking data 270obtained by a reference camera IMU 265) between a base reference camerapose and an updated reference camera pose, the base reference camerapose being associated with a base reference camera timepoint that occursprior to an updated reference camera timepoint.

Act 2604 of flow diagram 2600 includes identifying a match cameratransformation matrix. Act 2604 of flow diagram 2600 is performed, insome instances, using one or more processors 3405 of a computer system3400 (e.g., an HMD 200 and/or a user instrument 250). In someimplementations, the match camera transformation matrix is a 3Drotational matrix (e.g., identified based on inertial tracking data 240obtained by a match camera IMU 235) between a base match camera pose andan updated match camera pose, the base match camera pose beingassociated with a base match camera timepoint that occurs prior to anupdated match camera timepoint.

Act 2606 of flow diagram 2600 includes identifying an alignment matrix.Act 2606 of flow diagram 2600 is performed, in some instances, using oneor more processors 3405 of a computer system 3400 (e.g., an HMD 200and/or a user instrument 250). In some instances, the alignment matrixis based on visual correspondences between one or more reference framescaptured by the reference camera and one or more match frames capturedby the match camera

Identifying an alignment matrix according to act 2606 may includevarious acts. For example, flow diagram 2600 illustrates that act 2606Aincludes identifying a base matrix (it should be noted that a basematrix may refer to a previously computed alignment matrix). In someinstances, a computer system identifies a base matrix by identifying aset of base feature matches by performing feature matching between abase reference frame captured by the reference camera at the basereference camera pose and a base match frame captured by the matchcamera at the base match camera pose. The computer system also generatesa set of unprojected base feature matches by unprojecting the basefeature matches into 3D space. The computer system may then compute thebase matrix based on the set of unprojected base feature matches byminimizing a cost function (e.g., according to the Wahba method). Insome instances, the base matrix is a 3D rotation matrix.

Act 2606B includes identifying an updated matrix. In some instances, acomputer system identifies an updated matrix by identifying a set ofupdated feature matches by performing feature matching between anupdated reference frame captured by the reference camera at the updatedreference camera pose and an updated match frame captured by the matchcamera at the updated match camera pose. The computer system alsogenerates a set of unprojected updated feature matches by unprojectingthe updated feature matches into 3D space. The computer system may thencompute the updated matrix based on the set of unprojected updatedfeature matches by minimizing a cost function.

Act 2606C includes generating an aligned updated matrix. In someinstances, a computer system generates the aligned updated matrix byaligning the updated matrix with the base matrix using the basereference camera pose, the base match camera pose, the updated referencecamera pose, and the updated match camera pose.

Act 2606D includes fusing the aligned updated matrix with the basematrix. In some instances, a computer system fuses the aligned updatedmatrix with the base matrix by applying the aligned updated matrix andthe base matrix as inputs to a function for generating the alignmentmatrix. In some implementations, the function for generating thealignment matrix is an interpolation function, and the interpolationfunction may comprise a smoothness term that determines weights of thealigned updated matrix and the base matrix for generating the alignmentmatrix.

Act 2608 of flow diagram 2600 includes identifying a set of 3D points byunprojecting a set of pixels of the reference frame. Act 2608 of flowdiagram 2600 is performed, in some instances, using one or moreprocessors 3405 of a computer system 3400 (e.g., an HMD 200 and/or auser instrument 250). In some instances, a computer system identifiesthe 3D points by unprojecting the set of pixels of the reference frameusing an intrinsic matrix of the reference camera and using a uniformdepth.

Act 2610 of flow diagram 2600 includes generating a modified set of 3Dpoints. Act 2610 of flow diagram 2600 is performed, in some instances,using one or more processors 3405 of a computer system 3400 (e.g., anHMD 200 and/or a user instrument 250). In some instances, a computersystem generates a modified set of 3D points by applying the motionmodel to the set of 3D points identified according to act 2608.

Act 2612 of flow diagram 2600 includes projecting the modified set of 3Dpoints. Act 2612 of flow diagram 2600 is performed, in some instances,using one or more processors 3405 of a computer system 3400 (e.g., anHMD 200 and/or a user instrument 250). In some instances, a computersystem projects the modified set of 3D points (from act 2610) using anintrinsic matrix of the match camera. The projected modified set of 3Dpoints may enable a computer system to generate a composite image fordisplay to a user (e.g., on displays 225A and 225B of an HMD 200).

Updating Continuous Image Alignment of Separate Cameras

As described hereinabove, a system may generate a motion model that isconfigured to facilitate continuous mapping of sets of pixels from areference frame to a corresponding sets of pixels of a match frame, evenas the pose of the reference camera and the match camera change overtime (see, e.g., FIGS. 11-17 and Equations 2 and 4). A motion modelincludes an alignment matrix, which may be thought of as a runningaverage that uses a previous alignment matrix and an aligned currentmatrix to generate an updated alignment matrix for use in the motionmodel (as used herein, a “current matrix” may be thought of as analogousto an “updated matrix” as used herein). In some instances, as describedhereinabove, a computer system fuses a previous alignment matrix (orbase matrix) with an aligned current matrix to generate the updatedalignment matrix.

To illustrate, FIG. 27 depicts an example of a reference camera 260 of auser instrument 250 and a match camera 215 of an HMD 200 capturing anenvironment that includes a physical object 305 at different timepoints.Specifically, FIG. 27 illustrates the reference camera 260 capturing animage of the physical object 305 at a previous reference camera pose2705 and at a previous reference camera timepoint 2710. Similarly, FIG.27 illustrates the match camera 215 capturing an image of the physicalobject 305 at a previous match camera pose 2715 and at a previous matchcamera timepoint 2720.

Referring briefly to FIG. 28 , the images captured by the referencecamera 260 and the match camera 215 at their respective previoustimepoints form a previous frame pair 2820. In some instances, a systemidentifies visual correspondences 2815 between the images of theprevious frame pair 2820 (e.g., by performing feature matching on theimages of the previous frame pair 2820). A system may then determine aprevious alignment matrix 2810 using the visual correspondences 2815(and/or other components, such as one or more alignment matrices fortimepoints prior to timepoints associated with the previous frame pair2820, indicated by ellipsis 2825). In some implementations, the previousalignment matrix 2810 operates as part of a motion model to map pixelsof a previous reference frame (e.g., captured at the previous referencecamera timepoint 2710) to corresponding pixels of a previous match frame(e.g., captured at the previous match camera timepoint 2720) (e.g., seeEquation 1 and/or 2).

Returning to example(s) depicted in FIG. 27 , after capturing images ofthe physical object 305 with the reference camera 260 at the previousreference camera timepoint 2710 and with the match camera 215 at theprevious match camera timepoint 2720, the pose of the reference camera260 changes (indicated by arrow 2725) to correspond to the intermediatereference camera pose 2735 at the intermediate reference cameratimepoint 2740. Similarly, the pose of the match camera 215 changes(indicated by arrow 2730) to correspond to the intermediate match camerapose 2745 at the intermediate match camera timepoint 2750. Both thereference camera 260 and the match camera 215 capture images whileoriented according to their respective intermediate pose (e.g.,intermediate reference camera pose 2735 and intermediate match camerapose 2745, respectively) to form an intermediate frame pair.

As is evident from FIG. 27 , the reference camera 260, when orientedaccording to intermediate reference camera pose 2735, and the matchcamera 215, when oriented according to intermediate match camera pose2745, are directed away from the physical object 305 such that neithercamera captures the physical object 305. Accordingly, a computer systemmay fail to identify visual correspondences between the images of theintermediate frame pair. Thus, FIG. 28 illustrates that an intermediatealignment matrix 2830 for mapping pixels of an intermediate referenceframe (e.g., captured at the intermediate reference camera timepoint2740) to corresponding pixels of an intermediate match frame (e.g.,captured at the intermediate match camera timepoint 2750) mayincorporate the previous alignment matrix 2810, without generating anupdated alignment matrix based on the intermediate frame pair (e.g., seeEquation 2).

Returning again to example(s) depicted in FIG. 27 , after capturing theintermediate frame pair with the reference camera 260 at theintermediate reference camera timepoint 2740 and with the match camera215 at the intermediate match camera timepoint 2750, the pose of thereference camera 260 changes (indicated by arrow 2755) to correspond tothe current reference camera pose 2765 at the current reference cameratimepoint 2770. Similarly, the pose of the match camera 215 changes(indicated by arrow 2760) to correspond to the current match camera pose2775 at the current match camera timepoint 2780. Both the referencecamera 260 and the match camera 215 capture images while orientedaccording to their respective current pose (e.g., current referencecamera pose 2765 and current match camera pose 2775, respectively) toform an current frame pair 2850 (see, briefly, FIG. 28 ). As is evidentfrom FIG. 27 , the reference camera 260, when oriented according tocurrent reference camera pose 2765, and the match camera 215, whenoriented according to current match camera pose 2775, are directedtoward the physical object 305 such that both cameras capture thephysical object 305.

Referring again to FIG. 28 , in some instances, a system identifiesvisual correspondences 2845 between the images of the current frame pair2850 (e.g., by performing feature matching on the images of the currentframe pair 2850). A system may then determine a current matrix 2840using the visual correspondences 2845. FIG. 28 also illustrates that, insome instances, a system generates an aligned current matrix 2865 (e.g.,according to Equation 3) and fuses the aligned current matrix 2865 withthe previous alignment matrix 2810 (or the intermediate alignment matrix2830, which incorporates the previous alignment matrix) to generate anupdated alignment matrix 2860 (e.g., according to Equation 4). Forexample, with parenthetical reference to FIGS. 23A and 23B, a computersystem may utilize a fuser (e.g., fuser 2300) that implements aninterpolation function (e.g., interpolation function 2305), which, inessence, blends or combines the aligned current matrix 2865 (e.g.,aligned updated matrix 2210) with the previous alignment matrix 2810(e.g., base matrix 610 or alignment matrix 2315) to generate the updatedalignment matrix 2860 (e.g., alignment matrix 2315 or subsequentalignment matrix 2325).

In some implementations, the updated alignment matrix 2860 operates aspart of a motion model to map pixels of a current reference frame (e.g.,captured at the current reference camera timepoint 2770) tocorresponding pixels of a current match frame (e.g., captured at thecurrent match camera timepoint 2780) (e.g., see Equation 2).

As indicated hereinabove, in some implementations, an interpolationfunction utilizes a smoothness term (or smoothness function, seeEquation 4). In some implementations, the smoothness term determines howto weight the previous alignment matrix 2810 and the aligned currentmatrix 2865 for generating the updated alignment matrix 2860. By way ofexample, in some implementations, when a smoothness term has a valuethat approaches a maximum value (e.g., a value of 1), the interpolationfunction increasingly ignores the aligned current matrix 2865, givingmore weight to the previous alignment matrix 2810 for generating theupdated alignment matrix 2860, which may be beneficial when expectedaccuracy of the previous alignment matrix 2810 is high. Furthermore, insome implementations, when the smoothness term has a value thatapproaches a minimum value (e.g., a value of 0), the interpolationfunction increasingly ignores the previous alignment matrix 2810, givingmore weight to the aligned current matrix 2865 for generating theupdated alignment matrix 2860, which may be beneficial when the expectedaccuracy of the previous alignment matrix 2810 is low.

Different smoothness values may be appropriate for differentcircumstances. For example, selecting a smoothness value that gives moreweight to the previous alignment matrix 2810 may ameliorate the effectsof noise that may be present in the aligned current matrix 2865 (e.g.,noise introduced when performing feature matching). However, selecting asmoothness value that gives more weight to the aligned current matrix2865 may ameliorate potential inaccuracy of the previous alignmentmatrix 2810 with respect to current frames (e.g., inaccuracy broughtabout by IMU drift).

Selecting a smoothness value that gives more weight to the previousalignment matrix 2810 when the previous alignment matrix 2810 isinaccurate may cause inaccuracies in composite frames. Although theseinaccuracies may correct over time, the delay in providing accuratecomposite frames may render a motion model unsuitable for certainapplications (e.g., precise operations and/or operations with rapidchanges in camera orientation). Thus, in some implementations, it may bebeneficial to intelligently determine/update the smoothness value basedon the expected accuracy of the previous alignment matrix 2810.

The expected accuracy of a previous alignment matrix 2810 may depend onthe circumstances. For example, in some instances, a reference camera260 and/or a match camera 215 undergo(es) a significant amount of motionfrom the time that a previous frame pair 2820 was captured forgenerating a previous alignment matrix 2810, which may degrade theaccuracy/applicability of a previous alignment matrix 2810 with respectto frame pairs captured at current timepoints (e.g., because of IMUdrift, changes in parallax, etc.). This may occur, for example, where asystem fails to identify visual correspondences between images of one ormore intermediate frame pairs that intervene between current timepointsand previous timepoints at which visual correspondences weresuccessfully identified (e.g., as illustrated in FIGS. 27 and 28 ).

Accordingly, in some instances, a computer system intelligentlydetermines/updates the smoothness term based the expected accuracy ofone or more previous alignment matrices.

FIG. 29 illustrates the same frame capture and pose change sequencedescribed hereinabove with reference to FIGS. 27 and 28 . FIG. 29demonstrates that in some instances, a system identifies a differencevalue 2910 associated with the reference camera 260 between the currentreference camera timepoint 2770 and the previous reference cameratimepoint 2710 (which may be the most recent previous reference cameratimepoint for which visual correspondences were successfully identifiedto generate a previous alignment matrix). Additionally, oralternatively, a system may identify a difference value 2920 associatedwith the match camera 215 between the current match camera timepoint2780 and the previous match camera timepoint 2720 (which may be the mostrecent previous match camera timepoint for which visual correspondenceswere successfully identified to generate a previous alignment matrix).High difference value(s) 2910 and/or 2920 may indicate that an expectedaccuracy of a previous alignment matrix 2810 should be regarded as low(e.g., because of IMU drift errors accumulating), whereas low differencevalue(s) 2910 and/or 2920 may indicate that an expected accuracy of aprevious alignment matrix 2810 should be regarded as high. Additionaldetails concerning difference values will be provided hereinafter.

FIG. 30 illustrates an example of generating an updated alignment matrix3015 using an aligned current matrix 2865, a previous alignment matrix2810, and difference value(s) 2910 and/or 2920. For example, in someimplementations, a system uses the aligned current matrix 2865, theprevious alignment matrix 2810, and the difference value(s) 2910 and/or2920 as inputs to a fuser 3000 for generating the updated alignmentmatrix 3015. Similar to the fuser 2300 described hereinabove withreference to FIGS. 23A and 23B, the fuser 3000 may include aninterpolation function 3005 that incorporates a smoothness term 3010(and/or other alternative or additional terms, indicated by ellipsis3030).

The smoothness term 3010, in some instances, is intelligently determinedbased on the difference value(s) 2910 and/or 2920. As describedhereinabove, the smoothness term 3010 may determine a weight forblending the previous alignment matrix 2810 with the aligned currentmatrix 2865 to generate the updated alignment matrix 3015. For example,higher difference value(s) 2910 and/or 2920 may cause the smoothnessterm to approach a minimum value (e.g., a value of 0), which may causethe interpolation function 3005 to ignore the previous alignment matrix2810 to a higher degree when blending the aligned current matrix 2865with the previous alignment matrix 2810 to generate the updatedalignment matrix 3015. Conversely, for example, lower differencevalue(s) 2910 and/or 2920 may cause the smoothness term to approach amaximum value (e.g., a value of 1), which may cause the interpolationfunction 3005 to ignore the aligned current matrix 2865 to a higherdegree when blending the aligned current matrix 2865 with the previousalignment matrix 2810 to generate the updated alignment matrix 3015.

FIG. 31 illustrates example implementations of the difference value(s)2910 and/or 2920 that a system may identify in association with areference camera 260 and/or a match camera 215 at different timepoints.FIG. 31 illustrates that, in some instances, the difference value(s)2910 and/or 2920 is/are identified as a motion value 3105. For instance,a motion value 3105 may indicate an amount of motion 3110 that thereference camera 260 and/or the match camera 215 has/have undergonebetween a previous timepoint (e.g., previous reference camera timepoint2710, previous match camera timepoint 2720) and a current timepoint(current reference camera timepoint 2770, current match camera timepoint2780). A large amount of motion may indicate that IMU drift errors havebeen able to accumulate over time, indicating that the expected accuracyof a previous alignment matrix 2810 is low, which may cause a system toselect a low smoothness term 3010.

FIG. 31 also demonstrates that a motion value 3105 may indicate one ormore acceleration values 3115 experienced by the reference camera 260and/or the match camera 215 between a previous timepoint (e.g., previousreference camera timepoint 2710, previous match camera timepoint 2720)and a current timepoint (current reference camera timepoint 2770,current match camera timepoint 2780). High acceleration values may causeIMU drift errors to be higher than would exist for low accelerationvalues (e.g., particularly where components of the IMU become saturated,such a gyroscope), which may indicate that the expected accuracy of aprevious alignment matrix 2810 is low and may cause a system to select alow smoothness term 3010. The ellipsis 3125 indicates that a motionvalue 3105 may indicate other metrics, such as, by way of non-limitingexample, translational velocity (which may indicate a change in parallaxthat may render a previous alignment matrix 2810 less accurate).

As depicted in FIG. 31 , difference value(s) 2910 and/or 2920 areidentified as a temporal value 3120, such as a number of frames oramount of time that have/has elapsed since the most recent previoustimepoint at which visual correspondences were successfully identified.A high number of frames or amount of time between successful visualcorrespondences may indicate that a previous alignment matrix 2810 mayhave become inaccurate, which may cause a system to select a lowersmoothness term 3010. The ellipsis 3130 indicates that otherimplementations of difference value(s) 2910 and/or 2920 are within thescope of this disclosure.

Those skilled in the art will recognize, in view of the presentdisclosure, that the difference value(s) 2910 and/or 2920 may take onvarious forms. Accordingly, it will be appreciated, in view of thepresent disclosure, that the descriptions herein of difference values as“high” or “low” may refer to the magnitude of the difference value inabsolute terms (e.g., regardless of directionality and/or whether aparticular measured value has a positive or negative value).

The following discussion focuses on a particular example implementationof a smoothness term/smoothness function as described hereinabove. Onewill appreciate, in view of the present disclosure, that the followingparticular example implementation is provided to assist inunderstanding, and not by way of limitation.

In one example implementation, the function smoothness( ) is defined asfollows:

$\begin{matrix}{{{smoothness}{()}} = {{strength}*e^{\frac{{- 0.5}*{angle}\_{{sum}()}^{2}}{{falloff}^{2}}}}} & (5)\end{matrix}$

Where the parameters strength and the parameter falloff may be set topredetermined constants (e.g., 0.7 for strength, 30 for falloff). Thefunction angle_sum may be thought of as computing difference values forthe reference camera and the match camera as described hereinabove andmay be defined as follows:

angle_sum( )=angle(P _(ref_cur) ,P _(ref_last))+angle(P _(match_cur) ,P_(match_last))  (6)

Where P_(ref_cur) represents the current pose of the reference cameraand P_(ref_last) represents the reference camera pose at which visualcorrespondences were most recently successfully identified/computed(e.g., the reference camera pose for the most recent timepoint at whichan alignment matrix was successfully updated). Similarly, P_(match_cur)represents the current pose of the match camera and P_(match_last)represents the match camera pose at which visual correspondences weremost recently successfully identified/computed.

The function angle( ) may be generalized as follows:

angle(P,P′)=acos((P*(0,0,1)^(T))^(T)*(P′*(0,0,1)^(T)))  (7)

Which computes the cross product of the third columns of the twomatrices P and P′, which may correspond to the z axes of the twomatrices. The acos( ) operation may provide the angle between thecomputed axes.

FIG. 32 illustrates an example plot of an example implementation of asmoothness function described hereinabove with reference to Equations5-7. As is evident from FIG. 32 , the value of the smoothness term isgreater for smaller difference values (e.g., for smaller anglesumvalues, in absolute terms), and the value of the smoothness term issmaller for greater difference values (e.g., for larger anglesum values,in absolute terms).

Example Method(s) for Updating Continuous Image Alignment of SeparateCameras

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed.

FIG. 33 illustrates an example flow diagram 3300 depicting actsassociated with updating continuous image alignment of a referencecamera and a match camera. The discussion of the various actsrepresented in flow diagram 3300 includes references to various hardwarecomponents described in more detail with reference to FIGS. 1, 2, and 34.

Act 3302 of flow diagram 3300 includes identifying a previous alignmentmatrix. Act 3302 is performed, in some instances, using one or moreprocessors 3405 of a computer system 3400 (e.g., an HMD 200 and/or auser instrument 250). In some instances, the previous alignment matrixis associated with a previous frame pair captured at one or moreprevious timepoints by a reference camera and a match camera. In someinstances, the previous alignment matrix is based on visualcorrespondences between images of the previous frame pair.

Act 3304 of flow diagram 3300 includes identifying a current matrix. Act3304 is performed, in some instances, using one or more processors 3405of a computer system 3400 (e.g., an HMD 200 and/or a user instrument250). In some instances, the current matrix is associated with a currentframe pair captured at one or more current timepoints by the referencecamera and the match camera. In some instances, the current matrix isbased on visual correspondences between images of the current framepair. Furthermore, in some implementations, the current matrix is analigned current matrix, in that the current matrix is aligned with theprevious alignment matrix using inertial tracking data associated withthe reference camera and the match camera.

Act 3306 of flow diagram 3300 includes identifying a difference value.Act 3306 is performed, in some instances, using one or more processors3405 of a computer system 3400 (e.g., an HMD 200 and/or a userinstrument 250). The difference value may be associated with thereference camera or the match camera relative to the one or moreprevious timepoints and the one or more current timepoints. In someimplementations, the difference value comprises a motion valueassociated with the reference camera or the match camera relative to theone or more previous timepoints and the one or more current timepoints.The motion value may indicate an amount of motion associated with thereference camera or the match camera relative to the one or moreprevious timepoints and the one or more current timepoints.Additionally, or alternatively, the motion value may indicate anacceleration associated with the reference camera or the match camerarelative to the one or more previous timepoints and the one or morecurrent timepoints. In some implementations, the difference valuecomprises a temporal value associated with the reference camera or thematch camera relative to the one or more previous timepoints and the oneor more current timepoints.

Act 3308 of flow diagram 3300 includes generating an updated alignmentmatrix using the previous alignment matrix, the current matrix, and thedifference value. Act 3308 is performed, in some instances, using one ormore processors 3405 of a computer system 3400 (e.g., an HMD 200 and/ora user instrument 250). In some instances, a computer system utilizesthe previous alignment matrix, the current matrix, and the differencevalue as inputs to a fuser that includes an interpolation function forgenerating the updated alignment matrix. In some instances, theinterpolation function blends the previous alignment matrix with thecurrent matrix to generate the updated alignment matrix. Furthermore, insome implementations, the interpolation function comprises a smoothnessterm that is determined based on the difference value. The smoothnessterm may determine a weight for blending the previous alignment matrixwith the current matrix to generate the updated alignment matrix. Forexample, when the smoothness term has a minimum smoothness value, theweight may cause the interpolation function to ignore the previousalignment matrix when generating the updated alignment matrix.

Act 3310 of flow diagram 3300 includes using the updated alignmentmatrix as a previous alignment matrix for generating a subsequentupdated alignment matrix. Act 3310 is performed, in some instances,using one or more processors 3405 of a computer system 3400 (e.g., anHMD 200 and/or a user instrument 250). In some instances, a computersystem utilizes the updated alignment matrix, a subsequently obtainedcurrent matrix, and a subsequently identified difference value as inputsto a fuser that includes an interpolation function for generating thesubsequent updated alignment matrix (e.g., similar to the mannerdescribed hereinabove with reference to act 3308).

Example Computer System(s)

Having just described the various features and functionalities of someof the disclosed embodiments, the focus will now be directed to FIG. 34which illustrates an example computer system 3400 that may includeand/or be used to facilitate the embodiments described herein, includingthe acts described in reference to the foregoing Figures. In particular,this computer system 3400 may be implemented as part of a mixed-realityHMD, such as any HMD referenced herein.

Computer system 3400 may take various different forms. For example,computer system 3400 may be embodied as a tablet, a desktop, a laptop, amobile device, a cloud device, an HMD, or a standalone device, such asthose described throughout this disclosure. Computer system 3400 mayalso be a distributed system that includes one or more connectedcomputing components/devices that are in communication with computersystem 3400. FIG. 34 specifically calls out how computer system 3400 maybe embodied as a tablet 3400A, a laptop 3400B, or an HMD 3400C, but theellipsis 3400D indicates that computer system 3400 may be embodied inother forms as well.

The computer system 3400 includes various different components. FIG. 34shows that computer system 3400 includes one or more processors 3405(aka a “hardware processing unit”), a machine learning (ML) engine 3410,graphics rendering engine(s) 3425, a display system 3430, input/output(I/O) interfaces 3435, one or more sensors 3440, and storage 3445.

Regarding the processor(s) 3405, it will be appreciated that thefunctionality described herein can be performed, at least in part, byone or more hardware logic components (e.g., the processor(s) 3405). Forexample, and without limitation, illustrative types of hardware logiccomponents/processors that can be used include Field-Programmable GateArrays (“FPGA”), Program-Specific or Application-Specific IntegratedCircuits (“ASIC”), Application-Specific Standard Products (“ASSP”),System-On-A-Chip Systems (“SOC”), Complex Programmable Logic Devices(“CPLD”), Central Processing Units (“CPU”), Graphical Processing Units(“GPU”), or any other type of programmable hardware.

As used herein, the terms “executable module,” “executable component,”“component,” “module,” or “engine” can refer to hardware processingunits or to software objects, routines, or methods that may be executedon computer system 3400. The different components, modules, engines, andservices described herein may be implemented as objects or processorsthat execute on computer system 3400 (e.g. as separate threads).

The ML engine 3410 may be implemented as a specific processing unit(e.g., a dedicated processing unit as described earlier) configured toperform one or more specialized operations for the computer system 3400.The ML engine 3410 (or perhaps even just the processor(s) 3405) can beconfigured to perform any of the disclosed method acts or otherfunctionalities.

In some instances, the graphics rendering engine 3425 is configured,with the hardware processing unit 3405, to render one or more virtualobjects within the scene. As a result, the virtual objects accuratelymove in response to a movement of the user and/or in response to userinput as the user interacts within the virtual scene. The computersystem 3400 may include a display system 3430 (e.g., laser diodes, lightemitting diodes (LEDs), microelectromechanical systems (MEMS), mirrors,lens systems, diffractive optical elements (DOEs), display screens,and/or combinations thereof) for presenting virtual objects within thescene.

I/O interface(s) 3435 includes any type of input or output device. Suchdevices include, but are not limited to, touch screens, displays, amouse, a keyboard, a controller, and so forth. Any type of input oroutput device should be included among I/O interface(s) 3435, withoutlimitation.

During use, in some instances, a user of the computer system 3400 isable to perceive information (e.g., a mixed-reality environment) througha display screen that is included among the I/O interface(s) 3435 andthat is visible to the user. The I/O interface(s) 3435 and sensors3440/3465 may also include gesture detection devices, eye trackingsystems, and/or other movement detecting components (e.g., head trackingcameras, depth detection systems, gyroscopes, accelerometers,magnetometers, acoustic sensors, global positioning systems (“GPS”),etc.) that are able to detect positioning and movement of one or morereal-world objects, such as a user's hand, a stylus, and/or any otherobject(s) that the user may interact with while being immersed in thescene.

The computer system 3400 may also be connected (via a wired or wirelessconnection) to external sensors 3465 (e.g., one or more remote cameras,accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.). Itwill be appreciated that the external sensors include sensor systems(e.g., a sensor system including a light emitter and camera), ratherthan solely individual sensor apparatuses.

Storage 3445 may be physical system memory, which may be volatile,non-volatile, or some combination of the two. The term “memory” may alsobe used herein to refer to non-volatile mass storage such as physicalstorage media. If computer system 3400 is distributed, the processing,memory, and/or storage capability may be distributed as well.

Storage 3445 is shown as including executable instructions (i.e. code3450). The executable instructions (i.e. code 3450) representinstructions that are executable by the processor(s) 3405 of computersystem 3400 to perform the disclosed operations, such as those describedin the various methods. Storage 3445 is also shown as including data3455. Data 3455 may include any type of data, including image data,depth/disparity maps and/or other depth data, pose data, tracking data,and so forth, without limitation.

The disclosed embodiments may comprise or utilize a special-purpose orgeneral-purpose computer including computer hardware, such as, forexample, one or more processors (such as processor(s) 3905) and systemmemory (such as storage 3945), as discussed in greater detail below.Embodiments also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. Such computer-readable media can be any available media thatcan be accessed by a general-purpose or special-purpose computer system.Computer-readable media that store computer-executable instructions inthe form of data are one or more “physical/hardware computer storagemedia” or “physical/hardware storage device(s)” that are distinguishedfrom and that exclude mere transmission or transitory media. Incontrast, computer-readable media that merely carry computer-executableinstructions are “transmission media.” Thus, by way of example and notlimitation, the current embodiments can comprise at least two distinctlydifferent kinds of computer-readable media: computer storage media andtransmission media.

Computer storage media (aka “hardware storage device”) arecomputer-readable hardware storage devices, such as RAM, ROM, EEPROM,CD-ROM, solid state drives (“SSD”) that are based on RAM, Flash memory,phase-change memory (“PCM”), or other types of memory, or other opticaldisk storage, magnetic disk storage or other magnetic storage devices,or any other medium that can be used to store desired program code meansin the form of computer-executable instructions, data, or datastructures and that can be accessed by a general-purpose orspecial-purpose computer.

Computer system 3400 may also be connected (via a wired or wirelessconnection) to external sensors (e.g., one or more remote cameras) ordevices via a network 3460. For example, computer system 3400 cancommunicate with any number devices or cloud services to obtain orprocess data. In some cases, network 3460 may itself be a cloud network.Furthermore, computer system 3400 may also be connected through one ormore wired or wireless networks 3460 to remote/separate computersystems(s) 3470 that are configured to perform any of the processingdescribed with regard to computer system 3400.

A “network,” like network 3460, is defined as one or more data linksand/or data switches that enable the transport of electronic databetween computer systems, modules, and/or other electronic devices. Wheninformation is transferred, or provided, over a network (eitherhardwired, wireless, or a combination of hardwired and wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Computer system 3400 will include one or more communicationchannels that are used to communicate with the network 3460.Transmissions media include a network that can be used to carry data ordesired program code means in the form of computer-executableinstructions or in the form of data structures. Further, thesecomputer-executable instructions can be accessed by a general-purpose orspecial-purpose computer. Combinations of the above should also beincluded within the scope of computer-readable media.

Upon reaching various computer system components, program code means inthe form of computer-executable instructions or data structures can betransferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a network interface card or“NIC”) and then eventually transferred to computer system RAM and/or toless volatile computer storage media at a computer system. Thus, itshould be understood that computer storage media can be included incomputer system components that also (or even primarily) utilizetransmission media.

Computer-executable (or computer-interpretable) instructions comprise,for example, instructions that cause a general-purpose computer,special-purpose computer, or special-purpose processing device toperform a certain function or group of functions. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the embodiments may bepracticed in network computing environments with many types of computersystem configurations, including personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The embodiments may alsobe practiced in distributed system environments where local and remotecomputer systems that are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network each perform tasks (e.g. cloud computing, cloudservices and the like). In a distributed system environment, programmodules may be located in both local and remote memory storage devices.

One will also appreciate how any feature or operation disclosed hereinmay be combined with any one or combination of the other features andoperations disclosed herein. Additionally, the content or feature in anyone of the figures may be combined or used in connection with anycontent or feature used in any of the other figures. In this regard, thecontent disclosed in any one figure is not mutually exclusive andinstead may be combinable with the content from any of the otherfigures.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or characteristics. The described embodimentsare to be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

We claim:
 1. A system for updating continuous image alignment of areference camera and a match camera, the system comprising: one or moreprocessors; and one or more hardware storage devices storingcomputer-executable instructions that are executable by the one or moreprocessors to configure the system to: identify a previous alignmentmatrix associated with a previous frame pair captured at one or moreprevious timepoints by a reference camera and a match camera, whereinthe previous alignment matrix is based on visual correspondences betweenimages of the previous frame pair; identify a current matrix associatedwith a current frame pair captured at one or more current timepoints bythe reference camera and the match camera, wherein the current matrix isbased on visual correspondences between images of the current framepair; identify a difference value associated with the reference cameraor the match camera relative to the one or more previous timepoints andthe one or more current timepoints, wherein the difference valuecomprises a motion value associated with the reference camera or thematch camera relative to the one or more previous timepoints and the oneor more current timepoints; and generate an updated alignment matrix byusing the previous alignment matrix, the current matrix, and thedifference value as inputs for generating the updated alignment matrix.2. The system of claim 1, wherein the match camera is associated with ahead-mounted display (HMD), and wherein the reference camera isassociated with a user instrument.
 3. The system of claim 1, wherein thereference camera and the match camera are of a same camera modality. 4.The system of claim 1, wherein the current matrix is aligned with theprevious alignment matrix using inertial tracking data associated withthe reference camera and the match camera.
 5. The system of claim 1,wherein the reference camera and the match camera communicate with thesystem via a wireless link.
 6. The system of claim 5, wherein thewireless link comprises an ultra-wideband link.
 7. The system of claim1, wherein the motion value indicates an amount of motion associatedwith the reference camera or the match camera relative to the one ormore previous timepoints and the one or more current timepoints.
 8. Thesystem of claim 1, wherein the motion value indicates a rotationalacceleration associated with the reference camera or the match camerarelative to the one or more previous timepoints and the one or morecurrent timepoints.
 9. The system of claim 1, wherein the differencevalue comprises a temporal value associated with the reference camera orthe match camera relative to the one or more previous timepoints and theone or more current timepoints.
 10. The system of claim 1, whereingenerating the updated alignment matrix is performed by applying aninterpolation function that blends the previous alignment matrix withthe current matrix when generating the updated alignment matrix.
 11. Thesystem of claim 10, wherein the interpolation function comprises asmoothness term that is determined based on the difference value,wherein the smoothness term determines a weight for blending theprevious alignment matrix with the current matrix to generate theupdated alignment matrix.
 12. The system of claim 11, wherein when thesmoothness term has a minimum smoothness value, the weight causes theinterpolation function to ignore the previous alignment matrix whengenerating the updated alignment matrix.
 13. The system of claim 1,wherein the computer-executable instructions are further executable toconfigure the system to use the updated alignment matrix as a previousalignment matrix for generating a subsequent updated alignment matrix.14. The system of claim 1, wherein the computer-executable instructionsare further executable to configure the system to map a set of pixels ofa reference frame captured by the reference camera to a correspondingset of pixels of a match frame captured by the match camera using theupdated alignment matrix and inertial tracking data associated with thereference camera and the match camera.
 15. A system for updatingcontinuous image alignment of a reference camera and a match camera, thesystem comprising: one or more processors; and one or more hardwarestorage devices storing computer-executable instructions that areexecutable by the one or more processors to configure the system to:identify a previous alignment matrix associated with a previous framepair captured at one or more previous timepoints by a reference cameraand a match camera, wherein the previous alignment matrix is based onvisual correspondences between images of the previous frame pair;identify a current matrix associated with a current frame pair capturedat one or more current timepoints by the reference camera and the matchcamera, wherein the current matrix is based on visual correspondencesbetween images of the current frame pair; identify a difference valueassociated with the reference camera or the match camera relative to theone or more previous timepoints and the one or more current timepoints;and generate an updated alignment matrix by using the previous alignmentmatrix, the current matrix, and the difference value as inputs forgenerating the updated alignment matrix, wherein generating the updatedalignment matrix is performed by applying an interpolation function thatblends the previous alignment matrix with the current matrix whengenerating the updated alignment matrix.
 16. The system of claim 15,wherein the difference value comprises a motion value associated withthe reference camera or the match camera relative to the one or moreprevious timepoints and the one or more current timepoints.
 17. Thesystem of claim 16, wherein the motion value indicates an amount ofmotion associated with the reference camera or the match camera relativeto the one or more previous timepoints and the one or more currenttimepoints.
 18. The system of claim 15, wherein the interpolationfunction comprises a smoothness term that is determined based on thedifference value, wherein the smoothness term determines a weight forblending the previous alignment matrix with the current matrix togenerate the updated alignment matrix.
 19. The system of claim 18,wherein when the smoothness term has a minimum smoothness value, theweight causes the interpolation function to ignore the previousalignment matrix when generating the updated alignment matrix.
 20. Oneor more hardware storage devices storing computer-executableinstructions that are executable by one or more processors of acomputing system to configure the computing system to: identify aprevious alignment matrix associated with a previous frame pair capturedat one or more previous timepoints by a reference camera and a matchcamera, wherein the previous alignment matrix is based on visualcorrespondences between images of the previous frame pair; identify acurrent matrix associated with a current frame pair captured at one ormore current timepoints by the reference camera and the match camera,wherein the current matrix is based on visual correspondences betweenimages of the current frame pair; identify a difference value associatedwith the reference camera or the match camera relative to the one ormore previous timepoints and the one or more current timepoints, whereinthe difference value comprises a motion value associated with thereference camera or the match camera relative to the one or moreprevious timepoints and the one or more current timepoints; and generatean updated alignment matrix by using the previous alignment matrix, thecurrent matrix, and the difference value as inputs for generating theupdated alignment matrix.